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INTRODUCTION 



Few domestic programs have enjoyed federal support longer than has vocational education. 
Beginning with the Smith-Hughes Act of 1917, the federal government took an active interest in 
improving the quality of vocational education offerings by providing funds earmarked 
specifically for this part of the secondao^ and postsecondary curriculum. The Vocational 
Education Act of 1963 substantially increased federal support for vocational education and 
expanded federal concerns to issues of access as well as program quality. The recently enacted 
Carl Perkins Vocational Education Act of 1984 continues and, in many ways, strengthens the 
federal government's commitment to these twin goals of improved quality and improved program 
access. 

Despite this long history of support for and involvement in the vocational education 
enterprise, federal vocational education policy has generally been held in low regard, both by 
those critical of vocational education and desiring to change it and by those who defend its 
achievements and want them expanded. To the critics, federal policy has failed to change the 
enterprise for the better; to the defenders, federal policy has so "overregulated" the uses of 
federal funds that the money cannot be used effectively. While each group offers a different 
prescription for changing policy, both generally agree ihat existing policy is in need of a 
thorough overhaul. 

Vocational education policy thus poses an irony: while there is a broad and rather 
longstanding consensus that federal policy has been ineffective. Congress has continued, indeed 
frequently increased, funding for *he program — an action that makes sense only if the policy is 
working, when nearly everyone agrees it is not. How this strange set of affairs occurred and 
what should be done about it are the major concerns of this book. 

The failures of federal vocational education policy, we believe, stem from two major 
problems. First, those who make policy — legislators and their staff and the executive branch 
secretaries, under secretaries, and their staff — generally know little about vocational education. 
Most policy makers are products of the "academic" curriculum, and "vocational education" 
conjures up memories of wood shop, mechanical drawing, and vague images of some isolated 
wing of their high school where other students went off to work on cars, pound on typewriters, 
bang nails, raise pigs, or otherwise engage in activities that were not likely to become routine 
experiences for "die college-bound." Vocational education, even at the postsecondary level, has 
never been able to overcome this stereotype and convey to policy makers the complexity and 
diversity of the enterprise or the extent to which it touches the lives of young people. 

Second, and this problem has probably been compounded by the first, the general approach 
taken to vocational education policy has been heavily prescriptive, to the point of obsession with 
the procedural details of what may be done with federal funds for vocational education at the 
expense of attention to what should be and can reasonably be accomplished. Thus, policy has 
sought to impose sets of rather rigid, uniform procedures on an extraordinaiily diverse and 
higWy decentralized enterprise. That the enterprise has failed to conform too often has been seen 
as a "failure of vocational education" rather than a failure of the policy itself. 

In this book, we have two goals. The first is to convey to those who make policy — 
especially at the federal and state levels — a clearer understanding of vocational education and 
what can reasonably be expected of it. What we have to say will cut across the grain of much of 
the conventional wisdom about vocational education policy. For example, we will suggest that 
employment per se is uot the appropriate standard by which to judge the effectiveness of 
vocational education programs. Rather v/e argue that vocational education should be held 
accountable for making students employable and suggest some strategies for evaluating 
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cmployability in a specific, quantifiable fashion. This distinction between employment and 
employability is an important one with major implications for the directions federal policy must 
take if it is to become effective. 

Our second goal is to develop an approach to federal vocational education policy that is 
more performance-oriented and less preoccupied with process. Such a policy should reward 
results, while permitting a wide variety of strategies for obtaining these results. This is what we 
mean by moving "from prescriptive to permissive planning." For permissive planning to work, 
however, we m\x?x be clear about what we want from vocational education, so realizing the 
second goal clearly depends upon achieving the first. 

To these ends, we begin in Chapter One with a general overview of vocational education 
today — what is offered, who is served, what is accomplished, and what it costs. Chapter Two 
examines the evolution of federal vocational education policy from its beginnings in 1917 to the 
most recent reauthorization of the Vocational Education Act in 1984. Chapter Three develops our^ 
argument for adopting a new approach to federal vocational education policy and describes the 
elements of "permissive planning." Chapter Four then outlines a performance-based approach to 
evaluating vocational education programs and discusses how the desired outcomes of vocational 
education should be defmed and measured. Chapter Five addresses the problem of improving the 
accuracy and consistency of vocational education daja, an essential task if permissive planning 
and performance-based evaluation are to be adoptedJ Chapter Six presents thrfindings fronv^t- 
series of site visits to local secondary and postsecond^ institutions offering vocational 
education programs to elicit local perspectives on performance-based evaluation. Finally, Chapter 
Seven adds a concluding note on possible future directions for federal policy in vocational 
education. 
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CHAPTER ONE 
VOCATIONAL EDUCATION TODAY 



Vocational education in America is an enterprise of many contrasts, reflecting some of the 
very best and the very worst of what can be found in the nation's schools. In New York City, 
for example, the Murry Bergtraum School of Business and Commerce and Aviation High School 
are two of the finest secondary schools in the country. Annually, they turn away thousands of 
?pplications, and of those who are enrolled, over half go on to four-year college or university. 
At the postsecondary level, the Fashion Institute of Technology, part of the State University of 
New York, lures students whose options include Harvard, Princeton, and Yale. Combining 
rigorous vocational preparation for entering the fashion industry with solid liberal arts education, 
FIT is one of the most demanding and interesting postsecondary institutions anywhere in the 
world. 

But there is another side to vocational education, less glamorous and much less promising. 
In a major southern city, there arc high school automotive programs that do nothing more than 
teach students, all minority, how to wash and wax cars. In an adult training center in a large 
midwestem city, we have seen row upon row of idle machine tools, obsolete the day they were 
installed and useless for developing the skills demanded by local industries. In a large, poverty- 
ridden city on the west coast, a report by the NAACP Legal Defense fund noted that 44 percent 
of the federal vocational education funds provided by the federal government for programs for 
disadvantaged youth had never been spent ^ 

Because programs range so widely in quality, it is difficult to generalize about vocational 
education. As the late Henry David, Director of the Vocational Education Study for the National 
Institute of Education, was fond of saying: "Anything you say about vocational education is 
true."2 With that caveat, this chapter provides some general background on vocational education 
at the secondary and postsecondary levels. It describes vocational education as it exists today. 
What is offered as vocational education? Who is enrolled in vocational education? What has 
vocational education accomplished? W hat does it cost? 



WHAT IS OFFERED? 

In the United States, vocational education is carried out by some 15,700 comprehensive 
high schools, 1,400 area vocational schools or technical institutes, 1,100 community colleges, 
650 four-year postsecondary institutions, 550 correctional facilities, and 800 other non-degree 
granting public schools and technical institutes. In addition, there are over 6,800 private 
noncoUegiate postsecondary schools, commonly referred to as "proprietary schools," offering a 



^NAACP Legal Defense and Educational Fund, Inc., Vocational Education: Cause or Cure for Youth 
Unemployment, Washington, D.C.: 806 15lhe Street, N.W., April 1981, p.l6. 

^Dr. Henry David directed a five year study of vocational education for the National Institute of Education 
from 1977 to 1982. One of the authors had the privilege of working with Dr. David, while serving as Director for 
the Project o.i National Vocational Education Resources, a three year study contracted for by NIE to examine the 
distribution of federal, state, and local fiinds tor vocational education. 
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wide variety of job training programs.^ There is, then, no "sy stem" of vocational education in 
this country; rather there is a diverse conglomeration of public and private institutions, highly 
decentralized and enjoying substantial autonomy in governance, finance, and content of 
programs. 

In 1980-81, the latest year for which data are available, about 16.9 million students were 
enrolled nationwide in nine broad occupational areas of vocational education: agriculture, 
distribution and marketing, health, consumer and homemaking, occupational home economics, 
industrial arts, office occupations, technical, and trade and industrial.^ Within each of these nine 
areas, introductory programs are offered, and also a number of more advanced programs that are 
specific to particular occupations. Table 1 lists some of the specific program offerings within 
each general area. 

In 1980-81, there were about 10.5 million secondary students enrolled in vocational 
education. Given that there were about 12.5 million secondary school students enrolled that year, 
it would first appear that 85 percent of all secondary students were enrolled in vocational 
education, a startlingly high percentage if the data are accurate. Almost half of these secondary 
vocational students, however, were enrolled in consumer and homemaking programs and* 
industrial arts — programs that offer few, if any, job specific skills. Only 2.9 million, slightly 
more than one-fourth of all secondary vocational students, were enrolled in 11th or 12tii grade 
"occupationally specific programs," which purport to impart entry level job skills for a specific 
gainful occupation. Of the 6.3 million secondary students in grades 11 and 12, then, about 43 
percent were enrolled in occupationaUy specific vocational education programs.^ 

At the postsecondary level, the proportion of students enrolled in occupationally specific 
programs was much higher. Of the 6.4 million vocational postsecondary students, almost half 
were enrolled in these more advanced programs. Figure 1 summarizes the distribution of 
enrollment in occupationaUy specific progra^ns by program area. 

In addition to these regular vocational education programs, three other types of programs 
are operated under the aegis of vocational education: work-study, cooperative education, and 
apprenticeship. Under work-study, vocational education students may be employed in any public 
or non-profit agency, with their wages paid by vocational education funds (usually federal) rather 
than by the employer. The employment need not be related to the student's training, and the 
program serves more as a form of income support than a means of integrating classroom and on- 
the-job experience. The program enrolled fewer than 30,000 students in 1980-81. 

In contrast to work-study, cooperative vocational education programs are joint efforts 
between a school and an employer to provide a student with work experience directly related to 
training. For example, a student enrolled in commercial photography might spend half a day in 
school and four hours in the afternoon working as a laboratory assistant in a local film 
processing facility. The school and the employer negotiate a written agreement as to the nature 
and pace of the student's training. Cooperative education is highly regarded as an effective 
training method, but many vocational educators complain that it is expensive and cumbersome to 



"^National Center for Education Statistics (NCES), The Condition of Vocational Education, Washington, 
D.C: U.S. Government Printing Office, 1981, Table 2.1, p.8. 

^National Center for Education Statistics, Vocational Education Data System, 1980-81, Table 1 105, hereafter 
cued as VEDS. 

5VEDS, Table 1202. 




TABI E 1 

TYPICAL VOCATIONAL EDUCATION PROGRAM OFFERINGS 



Agriculture: Agricultural production, supplies and services, mechanics, products, hor.iculture, 
renewable natural resources, forestry, and fishing and fisheries. 

Distributive Education: Advertising, apparel and accessories, automotive, finance and credit, 
floristry, food distribution, food services, general merchandise, hardware and building materials, 
home furnishings, hotel and lodging, industrial marketing, insurance, personal services, real 
estate, recreation and tourism, and transportation. 

Health: Dental assisting, dental hygiene, dental laboratory technology, medical laboratory 
assisting, nursing, practical nursing, nursing assistance, rehabilitation, radiologic technology, 
mental health technology, inhalation therapy, medical assistant, community healtn aide, and 
medical emergency technician. 

Consumer and Homemaking: Home economics. 

Consumer and Homemaking (Occupational): Care and guidance of children, clothing 
management, food management, home furnishings and institutional and home management 

Industrial Arts: Woodshop, metalshop, and mechanical drawing. 

Business and Office: Accounting and computing occupations, computer and console operators, 
programmers, filing and office machines, personnel and training, stenographic and secre^^aial, 
supervisory and administrative management and typing. 

Technical: Architectural technology, automotive technology, civil technology, electrical 
technology, electronic technology, enrivonmental-control technology, industrial technology, 
mechanical technology, scientific data processing, commercial pilot training, fire and fire safety 
technology, police science technology, waste and waste water technology. 

Trade and Industrial: Air conditioning, appliance repair, body and fender repair, auto mechanics, 
automotive specialization, aviation occupations, commercial art, commercial phonography, 
carpentiy, electricity, masonry, plumbing and pipefitting, custodial services, diesel mechanics, 
drafting, foremans^ip, graphic arts, instrument maintenance and repair, maritime occupations, 
machine shop, machine tools, sheet metal, weeding and cutting, tool and die making, metallurgy, 
cosmetology, plastics, fireman training, law enforcement training, refrigeration, small engine 
repair, stationary energy sources, textile production and fabrication, upholstering, and 
woodworking. 
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FIGURE 1 

VOCATIONAL EDUCATION ENROLLMENTS 
BY PROGRAM AREA 1980-81 
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administer. Developing and maintaining the necessary contacts with employers is time 
consuming and labor intensive, because a single employer is rarely able to take more than a few 
students. Consequently, cooperative education programs are relatively small. Nevertheless, they 
are growing in popularity. From 1978-79 to 1980-81, during which time total vocational 
enrollments remained stable, enrollment in cooperative programs rose from 537,000 to 624,000, 
an increase of 16 percent in two ycars.^ 

Apprenticeship programs, which are one of the oldest forms of occupational training, are 
operated jointly by unions and employers who contract with public institutions to provide much 
of the training. Although students as young as sixteen are eligible, virtually all of the 181,000 
stu 'ents enrolled in apprenti^.eship programs in 1980-81 were in postsecondary institutions^ 

There are no national standards that govern the content of vocational programs, and even 
within states, programs sharing the same name may have little else in common. A program in 
''retail merchandising" in one school may do nothing more than train cashiers on electronic cash 
registers, while a program under the same name in a nearby area vocational school or community 
college may be preparing management trainees in such subjects as advertising, pricing, window 
display, and employee relations. 

To a large degree, the quality and rigor of a vocational education program depends on the 
type of institution in which it is offered. As a general rule, occupationally specific programs are 
offered in vocational high schools, area vocational schools, and community colleges, while 
introductory programs and courses in consumer and homemaking and industrial arts are offered 
in the comprenensi\e high schools. Unlike comprehensive high schools, which offer both 
vocational and general academic subjects, a vocational high school is a specialized school in 
which all or the majority of students a^e enrolled in vocational programs. An area vocational 
center pro /ides vocational instruction to students throughout a school district or group of 
districts. Students attend their home school for part of the day, where they receive general 
academic instruction, and go to to the area center for vocational education. Not all states, 
however, operate vocational high schools or area centers, in which case the complete secondary 
vocational program is offered througn the comprehensive high school. 

Many believe that programs in the vocational high schools, area vocational centers, and 
community colleges are generally superior to those in ±e comprehensive high schools. One, 
Gilbert Sev^all, writing iii Fortune Magazine, notes: 

Even \ all-purpose high schor c Jce job training seriously, they 
probably could not succ^.ed. Few st vocational budgets can stand the 

expense of high-auality, up-to-date induijial and clerical equipment. As a result, 
thousands of hig . school shops, filled with antiquated drill presses and rusting 
lathes, resemble industrial museums. M time when word-processing equipment 
is transforming office work, some 16-year-olds still peck at manual typewriters. 
Outmoded courses clutter th curriculum.^ 

The mo*^ specialized institutions can do a better job on several counts. They are usually 
able to offer a greater variety of vocational programs and provide much more in-depth 



^VEDS, Table 1601. 
^Ibid 

^Gilbert T. Sewall, "Vocational Education that Works/' Fortune, September 19, 198^^ p.70. 
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instruction. Often they locate close to a particular industry. A program in office skills located in 
the heart of a financial district, for example, or an aviation school adjacent to an airport, can 
provide stronger links between the schools and employers. Additionally, many of these schools 
enjoy more flexibility over staffing. California's Regional Occupation Programs and Centers, for 
example, rely heavily on part-time instructors hired under short-term contracts; consequently, 
they are better able to respond to changing labor market demands than comprehensive high 
schools where faculty are tenured Community colleges, while also having tenured faculty, make 
extensive use of part-time and short-term instructors.^ 

Despite these advantages, vocational education in comprehensive high schools has staunch 
defenders. In part, this support stems from the philosophical view expounded by the old Manual 
Arts Movement, holding that all students (or at least, all boys — the movement was strongest 
from about 1880 to 1930) need exposure to a broad-based industrial education program. Thus, 
Charles Bennett, one of the movement's chief spokesmen, wrote: 

Every man who would intelligently use the modem conveniences of his 
own home, or the labor-saving devices and conveniences of business life, must 
know something of the materials and principles of industry;... In fact, industrial 
development has been so rapid and so varied in our country — it has affected 
every man's life to such an extent that if he is to retain sufficient mastery of his 
environment to make it serve his needs, he is forced to acquire considerable 
practical knowledge of the materials, principles and processes of industry. And if 
the school is to furnish it, the school must be equipped with the tools of 
industry. 1^ 

Herein lies one of the raisons d'etre of the comprehensive high school, and many educators still 
subscribe to it. 

There is another reason some defend vocational education in the comprehensive high 
school: they fear that offering vocational education exclusively in vocational high schools or area 
centers segregates the vocational curriculum and makes it less accessible. Vocational education 
in this country has always suffered from a kind of second-class status in the education profession 
and in the public mind as well. In many high schools, vocational education, much like the 
general track, has been a dumping ground where students who are not going on to college simply 
mark time. Removing vocational education from the curriculum of the comprehensive high 
school, defenders fear, would simply exacerbate tracking and racial and social segregation.^^ 

Perhaps so, but given the track record of most vocational high schools and technical 
institutes, there is not much evidence to support this view. On this, Gilbert Sewall writes: 



^Charles S. Benson and £. Gareth Hoachlander, Descriptive Study of the Distribution of Federal, State, and 
Local Funds for Vocational Education: Final Report, Washington, D.C.: National Institute of Education, 1981; 
Charles S. Benson, *The Question of Quality,'* VocEd, Vol. 57, No. 6, pp. 27-29. 

l^Charles A. Bennett, The Manual Arts, as quoted in John Gallinelli, "Vocational Education Programs at the 
Secondary Level: A Review of Development and Purpose** Theodore Abramson et al. (ed.), Handbook of 
Vocational Education Evaluation, Beverly Hills: Sage Publicraons, 1979, pp. 31-32. 

^ ^Rupert Evans, **Why Vocational Education Belongs in the Comprehensive High School," VocEd, vol. 57, 
No. 6, pp. 24-26. 



But if a tour of the world of vocational ed offers many disappointments, it 
reveals some gratifying success stories as well. Quite surprisingly, a number of 
old-fashioned trade schools clustered in big cities and northeastern states may turn 
out to be the hot alternative schools of Uie 1980s... Another bright promise in 
vocational education is a new generation of technical institutes, which take in 
students with vared backgrounds: high school juniors and seniors ready to tackle 
a tough curriculum, high school graduates who choose the institutes instead of a 
community college, and iwhaps most importantly, people in their 20s who have 
had a bruising brash with low-paying jobs and come back to school to learn a 
trade. ^2 

The debate over the appropriate setting for vocational education is, of course, a part of a 
larger ongoing debate about the future of the American high school. It is difficult to predict what 
effect this larger debate will have on secondary education, bu this much can be said with 
certainty: unless it comes to grips with the wide range of quality in vocational offerings, and 
particularly the outdated and undemanding programs that dominate the comprehensive high 
school, it will ignore a major chunk of the secondary curriculum. 



WHO IS SERVED? 

In 1980-81, about 51 percent of students enrolled in vocational education were female, 76 
percent white, 16 percent black, 5 percent Hispanic, and 3 percent other minority.^^ With respect 
to race and sex, vocational education reflects almost exactly the demography of the general public 
school population. When viewed as a whole, there is no evidence that minorities or women have 
been shunted into this part of the non-college oriented curriculum. Among vocational education 
programs, however, there is considerable vanation in the distribution of students by race and 
sex. Black males, for example, are more concentrated in trade and industrial programs and 
industrial arts, while white males are more concentrated in agriculture and technical programs 
(See Figure 2). Relatively speaking, there are more white than black females in distributional 
programs, more black than white in occupational home economics. Women, generally, 
predominate in health, home economics, office, and consumer and homemaking, while men are 
more concentrated in agriculture, technical t adc and industrial, and industrial arts. Only 
distribution programs reflect a balance by sex (Sec Figure 3). To be sure, these patterns reflect 
inequalities in society at large, but they do explain why the elimination of race and sex 
stereotyping in vocational education has been such an issue. Aside from athletics, there is 
probably no other part of the school curriculum where race aiid sex stereotyping is more evident. 

Moreover, minorities and women are under represented in vocational high schools and area 
vocational schools. Minority students comprised about 28 percent of the enrollment in 
comprehensive high schools, compared to only 17 percent in the specialized schools. Similarly, 
53 percent of the enrollment in comprehensive high schools was female, compared to 40 percent 
in the specialized schools. Thus, to the extent that these schools offer superior and more 
advanced training than the comprehensive high schools, high quality programs appear to be less 
accessible to minorities and women. 



^^iibert T. Sewall, "Vocational Education that Works," Fortune, September 19, 1983, pp. 70-71. 
I^VEDS, Table 1105. 

^^National Center for Educational Statistics (NCES), The Condition of Vocational Education, Washington, 
D.C.: U.S. Government Printing Office, 1981, Tables 4.21 and 4.22, pp. 85-86. 



FIGURE 2 

DISTRIBUTION OF VOCATIONAL EDUCATION 
STUDENTS BY RACE AND SEX 
1980-81 



All Programs 
Agriculture 
Distribution 
Health 

Consumer & Homemaldng 
Occupational Home Econ. 
Indus^ Alts 

Office 
Technical 
Trade & Industdal 




20 



30 



40 50 60 
Pteioentage Distribution 



70 



90 



100 



Males 
I White, not Hispanic 
I Black, not Hispanic 
B Other Minority 



Females 

H White, not Hispanic 
Q Black, not Hispanic 
S Other Minority 



Source: VEDS, Table 1 105. 



ERLC 



1/ 

8 



FIGURE 3 

SEX DISTRIBUTION OF VOCATIONAL EDUCATION 
STUDx. ^ BY PROGRAM AREA 1980-81 




9 



Id 



During the last decade, much attention has also been devoted to making vocational 
education more accessible to han<^capped students. Data from NCES indicate that much remains 
to be acconr.plished on this goal. About 6.5 percent of the students between the ages of 14 and 
25 are handicapped, but in 1978-79, the vocational program with the highest reported percentage 
of handicapped students was consumer and homemaking, with 4.0 percent In office occupations 
and technical programs, 1.4 percent of the students were handicapped^^ 

How do vocational students compare with students in other parts of the curriculum on other 
characteristics? Data are spotty and available for secondary programs only. According to NCES, 
in 1978-1979 a slightly higher percentage of secondary students in vocational education 
programs lived in rural areas than did students in academic programs (31.8 percent of the total 
compared with 25.3 percent), although this difference seemed to be diminishing over time.^^ 
Compared to students in the academic track, vocational education students were also more likely 
to come from families whose parents had less formal education, and they typically had lower 
scores on standardized tests measuring verbal and mathematical competencies. In contrast to the 
comparison with the academic track, NCES rcpOTtcd few differences on these indicators between 
vocational education students and students enrolled in general cducation.l'^ 

These same data also suggest that differences between academic and vocational students 
may be diminishing. In 1972, the scores of vocational education students on tests of reading and 
mathematics were one and one-half to two standard deviations lower than those of academic 
students. By 1980, differences had dropped to about one standard deviation. The decline must 
be interpreted cautiously. It could reflect a decline in performance of students in the academic 
track, but it more likely reflects the fact the more able students are enrolling in superior vocational 
programs as they explore alternatives to college or seek occupational skills that will help them 
support their college educations in an era of rising costs and diminishing financial aid. 



WHAT IS ACCOMPLISHED? 

No other aspect of secondary education has been more studied with fewer conclusive 
findings than the effectiveness of vocational education. Reviewing the research on the impact of 
vocational education on various student outcomes — acquisition of skills, employment, 
earnings, and so forth — the National Institute of Education concluded: 

It should be emphasized that uie research results reported do not constitute, 
and should not be read as, an assessment of the effectiveness of either secondary 
or postsecondary vocational education programs. They are too limited — by both 
the data available for research and the difficulty of the research problem — to 



15lbid.,p. 62. 
^^ibid., p. 53. 
l^ibid., pp. 53-54. 

l^ibid., Tables 4.5 and 4.6, pp. 69-70. 
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attribute outcomes, both economic and noneconomic, to particular education 
experiences.^^ 

Much of the difficulty in obtaining conclusive findmgs about the effects of vocational 
education stems from widespread disagreement about what constitute appropriate outcomes by 
which to hold vocational education accountable. Many evaluators of vocational education tend to 
equate it with simple job training and adopt strict economic indicators (e.g., cost/benefit ratios or 
rates of return) of program performance. Most vocational educators argue that this approach 
takes too narrow a view. Unlike job training, which is primarily concerned with imparting an 
immediately marketable skill and appropriate woik attitudes, vocational education, promoters of 
this perspective assert, takes a broader and longer range view. In addition to imparting specific 
job skills, it is also concerned with the acquisition of basic skills in reading, writing, and 
computing, as well as more gcneralizable vocational skills that will serve students in a variety of 
ways as their occupational careers advance. Because the economic value of many of these skills 
cannot be known until well into the future (if even then strict) influences can truly be 
established), adopting simple cost/benefit standards for vocational education is not appropriate. 

Bearing in mind the difficulties surrounding evaluation of vocational education, what do we 
know about program outcomes? First, NCES reports that in FY 1979 about 46 percent of 1 1th 
and 12th grade vocational education students completed the programs in which they were 
enrolled, 46 percent remained in the program (many programs are of a two-year duration), 5 
percent left after completing more than half the progiam, and 3 percent dropped out of the 
program before completing half of the requirements. Among those who completed prograr s. 
Slightly more than half, 5 J .2 percent, were available for placement in the work force, 32 percent 
were not available for work either because they were attending school or for other reasons 
(enlisted in the military, for example), and the status of the remaining 17 percent was unknown. 
Among those available for work, 59 percent were employed in a field related to their training, 31 
percent were employed in another field, and 9 percent were unemployed but seeking worlc^o 

Do graduates of vocational education programs fare better in the job market than graduates 
from general education? The evidence on this question is mixed. The National Institute of 
Education cites studies that indicate that females enrolled in business and office programs 
experience less unemployment and enjoy higher weekly earnings than students in the general 
curriculum.2^ Another study indicated that, compared to students in the general curriculum, men 
enrolled in trade and industrial programs enjoy modest gains in income immediately after 
graduation from high school.22 Still another presented evidence that enrollment in vocational 
programs reduced drop-out rates. 23 With the exception of these findings, however, researchers 
have generally been unable to establish significant differences between the employment and 



^^National Institute of Education, The Vocational Education Study: The Final Report, Washington, D.C.: 
U.S. Department of Education, p. VII-22, 1981. 

^^National Center for Education Statistics, The Condition of Vocational Education, pp. 209-234. 

2lNational Institute of Education, The Vocational Education Study: The Final Report, pp. VIM to VII.9. 

22Robert H. Meyer, An Economic Analysis of High School Vocational Education, IV, The Labor Market 
Effects of Vocational Education, Washington, D.C.: The Urban Institute, 1981. 

23john T. Grasso and John R. Shea, Vocational Education and Training: Impact on Youth, Berkeley, Ca: 
Tlie Carnegie Council on Policy Studies in Higher Education, 1979. 




earning histories of students enrolled in vocational education a!id those enrolled in the general 
curriculum. 

This lack of evidence on positive impacts of vocational education must be interpreted 
carefully. First, except for reasons of cost, it is not clear why vocational education should be 
expected to deliver better performance than the general curriculum. Enrollment in vocational 
education as opposed to general education is largely a process of self-selection, with students 
making individual choices aboul v/hich type of program best suits their "spirations and learning 
styles. Vocational students might not perform as well as other students were this option not 
available to them. Unfortunately, this is a hypothesis that is exceedingly difficult to test. 

More importantly, the absence of positive findings does not mean that they do not exist. 
Designing adequate research models and securing accurate data have proven very difficult. 
Vocational education operates in such a diversity of settings, provides such a wide range of 
program offerings, and reflects such wide range quality, that controlled research is fraught 
with a number of unsolved theoretical and metho logical problems. Furthermore, no one has 
yet succeeded in obtaining good student foUow-t Jata that arc free of response bias and other 
inaccuracies, let alone providing sufficient detai* to differentiate among grade levels, delivery 
systems, costs of programs, and program types. Fast research has mostly relied on data sets that 
never were designed with the evaluation of vocational education in mind. In short, although a 
great deal has been spent on evaluation of vocational education urograms (federal law currently 
provides about $5 million annually for the National Center for Research in Vocational 
Education), the sad truth is that much of the research has been of very poor quality and that it has 
often failed to address issues of primary concern to setting national policy .^^ 



WHAT DOES IT COST? 

In 1980-81, the nation spent about $7.5 billion on vocational education, excluding 
expenditures by proprietary schools. About $850 million of this total, or 11.3 percent, 
represented federal funds, with the remainder supplied by state and local revenues. The federal 
figure is somewhat inflated, as only $753 million was allocated for 1980-81; states and local 
school district*; carried over $97 million in unspent federal funds from the previous year. Thus, 
the federal allocation now represents about one dollar in every ten expended for vocational 
education. 

Since 1972-73, state and local expenditures have grown about twice as fast as federal 
expenditures for vocational education. In 1980-81, state and local expenditures had increased 
160 percent from $2.2 billion in 1972-73 co $6.7 billion in 1980-81. Federal expenditures were 
up 77 percent from $482 million to $853 million (See Figure 4). 

Neariy two thirds of total expenditures are spent on secondary programs, where 
expenditures average $522 per student. Postsecondary expenditures per student average $426. 
Because no information is available on student contact hours, it is difficult to compare secondary 
and postsecondary expenditures per student. Secondary programs, however, do appear to have 
received a disproportionately large share of fiederal revenues; while 62 percent of state and local 
revenues for vocational education are expended on secondary programs, 80 percent of federal 
revenues are spent at that level (See Figure 5). 



2^Technassociatcs, Inc., An Evaluation of The National Center for Research in Vocational Education, The 
Ohio State University, Columbus, Ohio: a Report of an Expert Panel on Written Products Delivered between 
January 1978 and Jamtary 1982, a report submitted to the Planning and Evaluation Service, Office of Planning, 
Budget, and Evaluation, U.S. Department of Education, Washington, D.C., October 15, 1982. 
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FIGURE 4 

VOCATIONAL EDUCATION EXPENDITURES 
BY SOURCE OF FUNDS: FY 19^2:3 THROUGH FY 1980-81 
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Washington, D.C: U.S. Govenunent Printing Office, 1981, Table 7.1, p. 134; VEDS, Table 3102. 



FIGURE 5 
SECONDARY AND POSTSECONDARY 
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In 1980-81, the disbursement of federal funds for vocational education was authorized by 
the Vocational Education Act of 1963, as amended in 1968 and in 1976. The bulk of the federal 
funds were allocated under four major sections: 

1. Section 120: Basic Grants, $617 million in FY 1983; 

2. Section 130: Program Improvement and Support Services, $1 1 1 million; 

3. Section 140: Special Programs for the Disadvantaged, $15 million; 

4. Section 150: Consumer and Homemaking, $34 million. 



Additionally, in FY 1983, federal funds provided $22 million for programs of national 
significance (including the National Center for Research in Vocational Education) and $12 
million for state advisory councils and state planning. 

The states were required to match every federal dollar with one state or local oollar, a 
requirement that has litUe meaning now that the states are outspending the federal government by 
a factor of nine to one. Additionally, 10 r^rcent of t'le funds allocated under Sections 120 and 
130 had to be set aside for programs for handicapped students, and 20 percent for disadvantaged 
students. Federal law also directed the states to allocate federal funds to local school districts 
based on four factors: 1) relative financial ability, 2) concentrations of low-income families, 3) 
location in an economically depressed area, and 4) proposals for offering vocational education 
programs that are new to the area and designed to meet new and emerging manpower needs. 
These requirements were controversial, observed mainly in the breach. A nationwide study for 
the National Institute of Education found ro systematic relationship between the amount of 
federal vocational education funds allocated to local school districts and the factors specified by 
Congress for determining the distribution of funds. In short, efforts to target federal funds on 
particular types of districts, programs, or students with special needs have not been very 
effective. 



A SUMMING UP 

Vocational education in American today is a large and costly enterprise. Programs vary 
greatly in quality and relevance to labor market conditions, with programs in the comprehensive 
high schools being particularly suspect. Although vocational education as a whole mirrors the 
racial and sexual distributions of students throughout the secondary and postsecondary curricula, 
substantial variation exists among specific program areas. Moreover, there are strong indications 
that minorities and women have less access to the superior programs. The effectiveness of 
vocational education relative to the general curriculum is an issue that generates considerable 
controversy, with neither side of the debate able to put forth compelling evidence. Without 
question, many programs have been very successful at imparting basic and job specific skills and 
making students employable in skilled, reasonably high-paying, entry-level jobs. Precisely why 

^^Chorles S. Benson and E. Gareth Hoachlander, Descriptive Study of the Distribution of Federal, State, and 
Local Funds for Vocational Education, a repon prepared fcr the National Institute of Education, Washington, 
D.C., 1981. 
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they have been effective, however, is not well understood. Just as certainly, many programs are 
mediocre, if not scandalously outdated and undemanding. Why they have been allowed to remain 
so is also not well understood. 

There is much to be accomplished through a carefully crafted national policy on vocational 
education. Yet, with the exception of ii5> response to federal stimuli for expansion, the vocational 
education enterprise has been largely impervious to federal legislative prescriptions for 
improvement and change. Understanding the history of federal vocational education policy is the 
subject of the next chapter. 
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CHAPTER TWO 

THE EVOLUTION OF FEDERAL VOCATIONAL EDUCATION POLICY 



Since the adoption of the Smith-Hughes Act in 1917, the federal government has sought to 
improve the quality of the nation's vocational education enterprise, increase access to high quality 
offerings, and stiirulate economic development through the upgrading of occupational skills. 
From 1917 to 1963, federal policy provided modest support for rather narrowly defined 
occupational training programs in public high schools. Beginning with the Vocational Education 
Act of 1963, however, federal policy sought to expand vocational education. The act extended 
federal support for postsecondary programs and also sought to improve access to vocational 
education, especially among disadvantaged groups. 

The Vocational Education Act of 1963 underwent two major amendments. The first, in 
1968, expanded the definition of vocational education, established requirements for 
comprehensive planning and evaluation, and recmphasized improving the access of students with 
special needs. The second, in 1976, further expanded requirements for planning, evaluation, and 
accountability; called for the elimination of sex stereotyping in vocational education; and 
earmarked funding for programs for the disadvantaged, 1 andicapped, and students with limited 
English-proficiency. 

As federal legislation and regulations affecting vocational education became increasingly 
detailed, concern grew about their effectiveness. In 1981, completing a five-year comprehensive 
assessment of vocational education, the Vocational Education Study of the National Institute of 
Education concluded that the failure to focus federal policy on a few clearly stated objectives, the 
failure to design a variety of means to achieve a variety of ends, ani the failure to appreciate fully 
the limits of federal legislation on changing state policies and practices had severely undermined 
the effectiveness of the Vocational Education Act 

The Carl D. Perkins Vocational Education Act, enacted in October 1984, seeks to redress 
the most serious deficiencies of past vocational education legislation. It focuses on two specific 
goals for federal policy: improving program access and improving program quality. Over half the 
funds authorized for basic grants to the States are to be used for programs serving students with 
special needs — the handicapped, the disadvantaged, students with limited English proficiency, 
single parents and homemakers, adults in need of retraining, and men and women entering 
occupations nontraditional for their sex. The remaining funds are to be used exclusively for 
program improvement, innovation, and expansion. 



VOCATIONAL EDUCATION IN THE FIRST HALF OF THE CENTURY 

Federal vocational education policy has its rcots in the early 1900s. In 1906, a diverse set 
of actors representing industry, labor, and educators formed the National Society for the 
Promotion oflndustrial Education (NSPEE). During the next eight years, this group lobbied hard 
for federal aid to vocational educalion, and in 1914, President Woodrow Wilson appointed the 
Commission on National Aid to Vocational Education. He charged the commission with 
answering six questions: 

(1) To what extent is there a need for vocational education in the United 
States? 

(2) Is there a need for national grants stimulating the States to give 
vocational education? 
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(3) What Kinds or forms cf vocational education should be stimulated Dy 
national grants/ 

(4) How far can the Federal Government aid through expert knowledge 
vocational education in various States^? 

(5) To what extent should the Federal Government aid the States through 
national grants lor vocational education? 

(6) Under what conditions should giants to the States for vocational 
education be niade?^ 

Six months later the Commission re^rted: 

There is a great and crying need of providing vocational education of this 
character for every part of the United States — to conserve and develop our 
resources; to promote a more productive and prosperous agriculture, lo prevent 
the waste of human labor; to supplement apprenticeship; to increase the wage- 
earning powt* of our productive workers; to meet thz increasing demand for 
trained workmen; to offset the increased ::ost of Lving. Vocational education is 
therefore needed as a wise business investment for this Nation, because our 
national prosperity and happin^^s are at stake and our position in the narkets of 
the world cannot otherwise be maintains . ^ 

Federal legislation, however, was not forthcoming for another three years. Tnen, in 1917, with 
World War I straining the nauon's productive capacitxs. President Wilson signed the Smith- 
Hughes Act, initiating the federal government's involve^.r^t in vocational education. 

The Smith-Hughes Act provided $7.^ lilion aimually to promote vocational education in 
agriculture, trade and industrial education, and home economics in public secondary schools. It 
established a Federal Board for Vocational Education to administer the act and directed the states 
to create State Boards for Vocational Education. The states were fui Jier directed to prepare state 
plans describing what programs were provided and to make an annual report to the Federal 
Board on how federal money was spent In 1936, the George-Dean Act added an authorization 
for distributive education, bringing the total authorization to $14 million. In 1946, the Geoige- 
Barden Act added support for two youth organizations, the Future Farmers of America and 5ie 
New Farmers of America* Except for these two pieces of legislation, however, federal policy in 
vocational education was governed by the Smith-Hughes Act for about half a century.^ 

Compared to modem times, the aims of the Smith-Hughes Act were quite limited. It sought 
mainly to expand the supply of "'dlled labor in specific occupations such as practical nursing, the 



^Quoted in John Gallinelli, "Vocational Education Programs at the Secondary Level: A Review of 
Development and Purpose," in Theodore Abramson et ai.. Handbook of Vocational Education Evaluation, Beverly 
Hills, CA.: Sage Publications, 1979, p. 29. 

^Ibid., p. 29. 

^Lois-Ellin Datta, "Better Luck This Time: From Federal Legislation to Practice in Evaluating Vocational 
Education," in Theodore Abramson, Handbook on Vocational Edvcation Evaluation, pp. 41-42. 
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building trades, and defense-related occupations and to provide support for home economics. 
Federal funds were intended a stimulus for state efforts in these areas, and Congress paid 
close attention to how much nonfederal money was spent on vocational education. 

During the first three decades under Smith-Hughes, there is little evidence of dissatisfaction 
with federal vocational education policy. The traditional debate over whether vocational education 
properly belonged in the secondary school curriculum periodically waffed and waned (as it 
continues to do today), but until the late 1950s, there was little concern about the adequacy and 

effectiveness of the federal role. 

During the 1950s, however, three major developments produced increasing concern about 
federal education policy. First, the intensity of the Cold War generated serious concern about the 
nation* s ability to compete with Russia. Russia's launching of Sputnik in 1957 brought the issue 
to a hea^*, raising a national cry for overhauling mathematical, scientific, and technical training 
throughout the nation's secondary and postsecondary educational institutions. Second, tlie civil 
rights movement was gaining momentum, challenging longstanding inequities in the public 
schools. Issues of access and remediation to overcome centuries of neglect came to the public 
fore. Third, ttchnolcgy was playing an increasingly important role in the nation's economy. 
Accelerating technical change was generating demands for new kinds of skilled labor. Against 
these waves of change, the provisions of SmiUi-Hughes appeared increasingly inadequate. 

In 1958, Congress passed the National Defense Edi;cation Act Title VIE of this legislation 
provided funds for vocational education programs trainiMg technicians in occupations necessary 
for national defense. Three years later. Congress enacted the Area Redevelopment Act, which 
funded vocational education programs for unemployed and underemployed persons living in 
ecoi:onically depressed areas. Then in 1962, the Manpower Development and Training Act 
empowered local secondary and postsecondary school districts to provide vocaticaal education 
for unemployed trainees referred through the Department of Labor. Total appropriations for 
vocational education under all federal legislation rose to $79 million.^ 

With federal pobcy for vocational education becoming more and more dispersed in various 
pieces of legislation and among different federal agencies, the stage was set for a comprehensive 
review and consolidation. This was accomplished with enactment of The Vocational Education 
Act of 1963. 



THE VOCATIONAL EDUCATION ACT OF 1963 AND AMENDMENTS 

The Vocational Education Act of 1963 substantially expanded the federal government's role 
in vocational education. The legislation had t^"o major objectives, 1) to upgrade the rn-^ity of 
vocational education programs and 2) to improve opportunities for students with special needs 
who could not succeed in the regular vocational education programs. 

The first objective reflected the widespread view that vocational education suffered from 
low esteem. Over the years, vocational education had become increasingly isolated within the 
public high school. Early proponents of vocational education had been intent on reforming the 
American high school, broadening the curriculum to serve better the needs and aspirations of 
students less academically inclined than the college-bound. However, the rather narrow focus of 
Smith-Hughes on specific occupational training fostered a high degree of separation within the 
comprehensive high school. Shops were off in isolated parts of the building, and in some cities 



^Ibid., p. 42. 
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completely separate trade schools had been established for vocational education. This geographic 
isolation, combined with a general lack of interest in vocational education on the part of college- 
bound students, tended to relegate vocational educati'^n to second cliss status. In many high 
schools, a vocational track emerged that did not enjoy the esteem of the academic track and fell 
well short of the high hopes of tum-of-the-century visionariej^ • no sought a sense of equal 
worth between the liberal and the manual arts. 

The means employed to achieve the upgrading and expansion of vocational education were 
two-fold. First, federal appropriations were greatly increased and leveraged by state/local 
matching. The act authorized appropriations of $60 million for FY 1964, $1 18.5 million for FY 
1965, $177.5 million for FY 1966, and $225 million for FY 1967 and each year thereafter. The 
states were required to match each federal dollar with at least one state or local dollar. Second, 
one-third of each state's allotment was to be used either for construction of area vocational 
schools or for postsecondary vocational programs.^ Thus, it was hoped that specialization and 
association with postsecondary institutions would enhance the prestige and influence of 
vocational education programs. 

To improve the access of students with special needs. Section 4 (a) of the act stipulated that 
federal funds under the states' basic alloment could be used to fund "vocational education for 
persons who have academic, socio-economic, or other handicaps that prevent them from 
succeeding in the regular vocational education program." The legislation did not require the states 
to spend any minimum amount for this purpose, but did provide $30 million to implement work- 
study programs that provided compensation for full-time students, age 15 to 21, who would 
otherwise be unable to participate in vocational programs without financial support 



THE 1968 AMENDMENTS 

In 1968, Cong^ss enacted a number of important revisions to the Vocational Education 
Act. First, it broadened considerably the definition of vocational education, further blurring the 
distinction with academic studies. Henceforth, vocational education included not only training of 
occupational skills H*^^ also 'instruction related to the occupation or occupations for which the 
students are training or instruction necessary for students to benefit from such training."^ Hence, 
instruction in basic math and reading skills, as well as remedial instruction, was brought under 
the rubric of vocational education and became eligible for federal funds. In addition, the 
amendments explicitly included industrial arts.^ 



^An area vocational school is a specialized high school that is used exclusively or principally for the 
provision of vocational education. Students typically attend the area vocational school for part of the day, 
receiving their academic instruction at their home high school. 

^P.L. 90-576. Sect. 108 (1). 

^Whether industrial arts constitutes vocational education has long been a controversial issue among vocational 
educators. Industrial arts was originally conceived by its proponents as an integral part of general education — 
exposing students (until recendy, mosdy boys) to industrial skills regardless of their occupational goals. Thus, it 
was not intended to prepare students for a specific occupation or trade but rather to familiarize students with the 
"culture'* of the industrial world Such exposure was considered as valuable for the prospective banker or physician 
as for the prospective carpenter or electrician. Vocational education, on the other hand, has always had as its main 
goal preparing students for work, and many vocational educators regard industrial arts as lacking the rigor and 
complexity necessary to impart useful skills thai make one employable in the labor market. They resent, 
therefore, the eligibility of industrial arts for federal funds intended to improve vocational education. 
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Second, the 1968 Amendments stipulated that specific portions of the basic allotment be 
spent on programs for students with special needs. Twenty-five percent of each state's basic 
grant was to be spent for "vocational education for persons (other than handicapped persons 
defined in section 1086) who have academic, socioeconomic, or other handicaps that prevent 
them from succeeding in the regular vocational education program."^ An additional 10 percent of 
the basic grant was to be reserved for programs for handicapped students. 

Third, the legislation expanded the requirements for planning and reporting. It created a 
National Advisory Council on Vocational Education and directed each state to create a State 
Advisory Council to evaluate vocational education programs, services, and other activities 
receiving federal support 

Finally, the legislation continued federal support for work-study programs and also 
authorized appropriations (starting at $20 nullion in FY 1969 and rising to $75 million in FY 
1975) for cooperative vocational education programs. These progranfis were intended to combine 
work experience with formal classroom education and to improve interaction between employers 
and educators. 

What impact did the Vocational Education Act have during its first ten years? Good data are 
difficult to come by, but at least one fact is clear: vocational education programs expanded 
enormously. In 1964, there were about 4.6 million students enrolled in vocational education. 
Federal expenditures amounted to just over $55 million, and state and local governments spent 
about $278 million. By 1974, enrollments were almost 13.8 million, up 300 percent from a 
decade earlier. Federal expenditures had risen to $468 million, more than an eight-fold increase, 
and expenditures by state and local governments were nearly $3 billion, more than 10 times 
expenditures in 1964.^ Even tempered for the effects of inflation and growth in secondary and 
postsecondary enrollments generally, this growth is impressive. There is no doubt that the 
objective of program expansion was achieved. 

On other aims, however, success was less apparent In 1974, the General Accounting 
Office issued a highly critical report on vocational education and the use of federal funds. The 
report charged that states were distributing federal funds to local school districts without regard 
to need and without giving priority to efforts for program improvement or expansion; the bulk of 
federal funds was being used simply to maintain existing programs, many of poor quality. 
Moreover, the report found that states were not complying with the matching requirements for 
federal funds set aside for handicapped and disadvantage! students. Although most states far 
exceeded the one to oiie matching requirement over all, often by a factor of eight or nine to one, 
several of the states examined spent less than one state and local dollar for every federal dollar on 
programs for students with special needs. Generally, the report concluded, improving the access 
of students with special needs to high quality vocational education programs had not received 
high priority. The report also noted that sex stereotyping was pervasive throughout the vocational 
education curriculum. 

8pi. 90-576. Section 122 (a) (4) (A). 

^National Center for Education Statistics. Digest of Education Statistics 198L Washington, D.C.: U.S. 
Government Printing Office. Tables 15i and 154. pp. 165-166. 

^^U.S. General Accounting Office. What is the Role of Federal Assistance for Vocational Education?, Repon 
of the Controller General to Congress. Washington. D.C.: U.S. Government Printing Office, December 1974. 
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The GAO report was based on an in-depth analysis of vocational education in only seven 
states, and the report was quickly attacked as unrepresentative by the Amencan Vocational 
Sa^on a^d £e Bureau of Occupational and Adult Education (BO AE the feder^^^^^^^^^ 
charged with administering the Vocational Education Act within the then U.S. Office ot 
Education.!' Subsequent congressional hearings, however, as well as visits by committee staff 
fo Sonal states? supported the findings of the GAO report. It figured prominenUy m the 
development of the 1976 Amendments to the Vocational Education Act. 12 

THE 1976 AMENDMENTS 

By 1976 one of the major objectives of federal vocational education policy, program 
expansion, had been achieved. In that year cnroUments in vocational education programs stood at 
15 3 million students, and the states were spending more than $4.1 bilhon, nearly eight times 
federal expenditures of $534 million. I3 ConsequenUy, the PoHcy focus shifted mainly to issues 
of equity and program quality in vocational education. The 1976 Amendments sought to improve 
equity and program qualit^ by relying on three strategies: 1) increased regulation of the 
Xedures used to distribute f^cral finds to local school districts, 2) efforts to reduce sex 
stereotvDine and 3) greater regulation of planning and evaluation processes, mcludmg the 
SSent of a national Vocational Education Data System (VEDS) to provide comparable 
data on program enrollments, staff, expenditures, and outcomes. None of these strategies proved 
to be very effective. 

Funds Distribution Procedures 

With respect to funds distribution, the legislation and subsequent regulations carried 
forward the past provisions of setting aside portions of each state's basic aliotmem to support 
programs foV handicapped studems and studems who were econormcally or academically 
disldvantaged, including students with limited English proficiency Under the 976 
amendments, however, thise "setaside" funds could be used to fund only the excess costs of 
suTh programs, that is program costs above the average expenditure per student m vocational 
education for students who were not handicapped or disadvantaged. 

In addition to the setaside requirements, the 1976 Amendments directed the states to base 
the allocation of funds to LEAs on four factors: 1) location in an econoniically depressed area, 2) 
pr^^sSor offering vocational education tiiat are new to the area ^d ^'Zf.T^ltK rl^^ 
emSging manpower needs, 3) the relative financial abiUty of ihe LEA, and 4) an LEA s relative 
concentration of low-income families or individuals. Specifically, the legislation stated. 

llUnder the new U.S. Department of Education, the Bureau has been reorganized and renamed. It is now *e 

Office of vocational and Adult Education (OV AE). It is a change in name only ^owev^ Hi«onc^ 

hi beem more a defender and spokesperson for the vocational educaaon community than an independent, objecuve 

overseer of federal programs. 

12see tt.e National Institute of Education. The Vocational Education Study: The Final Report, pp. 1-3 to 1- 



4. 



13National Center for Education Statistics. Diiest of Education Statistics 1981 . Tables 151 and 154, pp. 165- 

1^. 
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that the State shall in considering the approval of such applications (for 
funds) give priority to those applicants which — 

(i) arc located in economically depressed areas and areas with high rates of 
unemployment, and are unable to provide the resources necessary to meet the 
vocational education needs of those areas without Federal assistance, and 

(ii) propose programs which are new to the area to be served and which are 
designed to meet new and emerging manpower needs and job opportunities in the 
area, and, where relevant, in the State and the Nation. 

Additionally, the Congress specified that: 

the State shall, in determining the amount of funds available under the Act 
which shall be made available to those applicants approved for funding, base such 
distribution on economic, social, and demographic factors relating to die need for 
vocational education among various populations and the various areas of the 
State, except that — 

(i) the State will use as the two most important factors in determining this 
distribution (I) in the case of local educational agencies, the relative fmancial 
ability of such agencies to provide the resources necessary to meet the need for 
vocational education in the areas they service and the relative number or 
concentration of low-income families or individuals within such agencies, and (11) 
in the case of other eligible recipients, the relative financial ability of such 
recipients to provide the resource to initiate or maintain vocational education 
programs to meet the needs of their students and the relative number or 
concentration of students whom ihey serve whose education imposes higher than 
average costs, such as handicapped students, students from low-income families, 
and students from families in which English is not the dominant language.^^ 

The Congress explicitly prohibited distributing funds to eligible recipients in equal amounts 
per student or by reimbursing a uniform percentage of expenditures for vocational education: 

The State will not allocate such funds among eligible recipients within the 
State on the basis of per capita enrollment or through matching of local 
expenditures on a uniform percentage basis. 

In short the general aim of the Congress with respect to the distribution of federal funds 
was quite clear. States were expected to focus their resources in LEAs with high concentrations 
of students with special needs and in LEAs with low wealth or other indications that they were 
economically depressed. Riority was to be given to new and innovative programs. How this 
was to be accomplished, however, was not addressed. The legislation provided no definitions 
for such terms as "economically depressed area," "new program," "priority," "relative financial 



l^P.L. 94-482, Sec. 106(a) (5) (A). 
^^P.L. 94-482. Sec. 160 (5) (jd) (i). 
l^P.L. 94-482. Sec. 160 (5) (B) (ii). 
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ability," or "relative concentration." Although the law clearly stated that uniform per capita 
distributions of funds would not be tolerated, it gave no indication as to how close a state might 
come to a uniform distribution and still be in compliance. Moreover, the act did not stipulate a 
formula for funds allocaiion or indicate that states would be required to design a formula for 
distributing funds. 

These funds distribution requirements generated much controversy and confusion. The task 
of interpreting the legislation and issuing regulations fell to BOAE, which lacked the technical 
expertise to design and evaluate funds distribution procedures. In some instances, BOAE 
approved distribution formulas that made no sense mathematically and statistically and in other 
cases disallowed perfectly sensible designs.^*^ After reviewing the procedures used in each of the 
50 states during 1978-79, the second year under the 1976 Amendments, the Project on National 
Vocational Education Resources (PONVER) reported to the National Institute of Education: 

While some states clearly demonstrated better understanding of the 
complexities of formula design than others, no state was using a procedure free of 
technical difficulties, arbitrary judgments, unexplained calculations, questionable 
interpretations of federal law, or inaccurate and inappropriate data.^^ 

In 1979, BOAE circulated for the third time a revised draft of proposed funds distribution 
procedures, but like the two that preceded it, this one was fraught with errors and ambiguities. 
BOAE fmally abandoned the attempt to issue a standard policy guide and continued to review 
funds distribution procedures on a state-by-statc basis. 

Even had BOAE been able to issue clear guidelines, the states would have encountered 
difficulties in complying with the funds distribution directions of the 1976 Amendments. For one 
thing, two of the criteria governing funds distribution were potentially contradictory. The states 
were directed to target funds to areas that were "economically depressed" and to areas with 'Sew 
and emerging manpower needs." These areas were not likely to be one in the same. Indeed, aieas 
with new and emerging manpower needs were more likely to be economically vigorous 
communities with high rates of growth and low rates of unemployment — areas the least in need 
of federal assistance. One criterion could well offset the other, producing the very uniform 
distribution that the Congress explicitly prohibited 

There were other difficulties as well. The notion of "excess costs" caused numerous 
problems. Most LEAs, especially the smaller ones, do not maintain accounting sys .ems that 
make it possible to identify expenditures for handicapped and disadvantaged students 
"mainstreamed** in the traditional classroom. The regulations permitted LEAs to count the full 
costs of separate programs for handicapped and disadvantaged students, a requirement within the 
capability of most LEAs accounting procedures, but this simply encouraged LEAs to develop 
separate programs in direct contradiction of instructions to mainstream whenever possible: 



^"^E. Gareth Hoachlander» "Disuibuting Federal Categorical Aid by Formula: The Case of Vocational 
Education," Stanford, California: Institute for Research on Educational Finance and Governance. Stanford 
University, 1981. 

^^Charles S. Benson and E. Gareth Hoachlander, Descriptive Study of the Distribuiion of Federal, State, and 
Local Funds for Vocational Education: Final Report, Washington, D.C.: National Institute of Education, 1981, 
p. 130. 
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The State shall use, to the maximum extent possible, the funds expended for 
handicapped and disadvantaged persons to enable these persons to participate in 
regular vocational education programs.^^ 

In light of these and other problems, it is not surprising that the actual distribution of 
federal dollars to LEAs bore little relation to the aims of Congress. PONVER analyzed the 
allocation of funds to secondary and postsccondary LEAs in a sample of twelve states. As one 
method of testing state compliance, regit sslon analysis was used to examine differences in the 
amounts of federal funds allocated to LEAs. If states were meeting federal objectives, then, other 
things being equal, one would have expected to find larger allocations in LEAs with'low wealth, 
high concentrations of handicapped and disadvantaged students, high unemployment, and 
concentrations of low income families or individuals. 

PONVER' s analysis generally revealed no statistically significant relations between funding 
levels and the various factors Congress specified for determining allocations. In only one of the 
twelve states was spending consistently higher in LEAs with lower relative financial ability. A 
statistically significant positive relationship between funding levels and concentrations of low- 
income families was found in only four of the twelve states. Positive relationships between 
funding levels and concentrations of handicapped students existed in three of eleven states (data 
were unavailable for one), with concentrations of disadvantaged students in t^. ur of nine states, 
and with rate of unemployment in one of the twelve states.^ 

In short, because clear guidelines for distributing funds were absent, because BOAE lacked 
the expertise to evaluate state procedures, and because certain provisions of the 1976 
Amendments were contradictory or not able to be implemented, the states enjoyed virtually 
unlimited discretion in the distribution of federal funds. In 1981, when the new administration 
took control and reorganized BOAE into the Office of Vocational and Adult Education (OVAE), 
the effort to design clear guidelines and procedure^ lor funds allocation quietly ceased. For all 
practical purposes, allocation of the basic allotment under the Vocational Education Act operated 
like a block grant 



Funds Distribution and The Elimination of Sex Stereotyping 

The 1976 Amendments added a major new objective to federal vocational education policy 
— overcoming sex discrimination and sex stereotyping. The legislation was replete with 
language encouraging the states to give high priority to this aim. Thus, the Act added to the 
opening declaration of purpose that federal funds were to be used: 

to develop and cany out such programs of vocational education within each 
State so as to overcome sex discrimination and sex stereotyping in vocational 
education programs (including programs of homemaking), and thereby furnish 
equal educational opportunities in vocational education to persons of both 
sexes. 21 

^^Federal Register. Vol 43., No. 59, Monday, March 22, 1978, p. 12357. 

^^Charles S. Benson and E. Gareth Hoachlandcr, Descriptive Study of the Distribution of Federal, State, and 
Local Funds for Vocational Education: Final Report, pp. 137-183. 
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The legislation further directed the states to: 



assign such full-time personnel as niay be necessary to assist the State board 
in fulfilling the purposes of this Act by — 

(A) taking such action as may be necessary to create awareness of programs 
and activities in vocational education that are designed to reduce sex stereotyping 
in all vocational education programs; 

(6) gathering, analyzing, and disseminating data on the status of men and 
women, students and employees in the vocational education programs of that 
State; 

(C) developing and supporting actions to correct any problems brought to 
the attention of such personnel througn activities carried out under clause (B) of 
this sentence; 

(D) reviewing the distribution of grants by the State board to assure that the 
interests and needs of women are addressed in the projects assisted under this 
Act; 

(E) reviewing all vocational education programs in the State for sex bias; 

(F) monitoring the implementation of laws prohibiting sex discrimination in 
all hiring, firing, and promotion procedures with the State relating to vocational 
education; 

(G) reviewing and submitting recommendations with respect to the 
overcoming of sex stereotyping and sex bias in vocational education programs for 
the annual program plan and report; 

(H) assisting local educational agencies and other interested parties in the 
State in improving vocational education opportunities for women; and 

(I) making readily available to the State board, the State and National 
Advisory Councils on Vocational Education, the State Commission on the Status 
of Women, the Conunissioncr, and the general public, information developed 
pursuant to this subsection.^ 

Each state was directed to reserve $50,000 annually from its federal allotment to carry out these 
activities. 



Despite the amount of language devoted to the goals of elintinating sex stereotyping and 
improving programs for handicapped and disadvantaged students, the 1976 Amendments offered 
little in the way of specific directions as to how this was to be accomplished. It is therefore riot 
surprising that activities to promote sex equity were uneven. As part of its survey of 1,200 LEAs 
in a representative sample of ten states, PONVER asked LEAs about their efforts to promote sex 
equity in vocational education. Districts were asked if they had, during the current academic year 
(1979-80), spent any federal, state, or local funds on special activities to promote sex equity in 



22p.L. 94-482, Sec. 104(b)(1). 



vocational education. Examples of special activities included assemblies, speakers, films, and 
workshops. Among the secondary LEAs, 22 percent reported that they had expended funds for 
sex equity. At the postsecondary level, 40 percent reported doing so.^^ 

The proportion of LEAs undertaking activities to eliminate sex stereotyping varied 
according to size of community. In large cities, a majority of secondary school districts reported 
carrying out such activities; almost 60 percent of districts in these urban areas said they had spent 
funds to promote sex equity. In rural areas, on the other hand, only 10 percent of the districts 
reported spending money to reduce sex stereotyping. The proportion of suburban districts 
reporting expenditures for sex equity fell between these two extremes. Thus, 1 7 percent of LEAs 
in small suburban conmiunities reported expenditures for sex equity, and 33 percent of LEAs in 
mid-size cities (population between 50,000 and 100,000) reported doing so. 

These results need to be interpreted carefully. On the one hand, three quarters of the 
districts responding to the survey reported no activities to promote sex equity. Moreover, among 
those reporting some activities, in half of these expenditures amounted to less than $500 during 
the academic year. These figures indicate a rather low level of activity and suggest that federal 
policy had not had much impact. On the other hand, sex equity activities were much more 
prevalent in the larger districts, indicating that while only 25 percent of the LEAs were exposed 
to sex equity issues, a much larger percentage of students were affected, perhaps as many as 40 
to 50 percent. When one considers that the survey was conducted only two years after the 
implementation of the 1976 Amendments, this level of activity is significant But has the sex 
composition of vocational education programs been affected? There is some evidence to suggest 
that it has. 

In 1978-79, 52 percent of the students enrolled in health, consumer and homemaking, and 
office occupations were female. In contrast, more than 75 percent of the students enrolled in 
agriculture, industrial arts, technical, and trade and industrial programs were male. Of the 
program areas dominated by females, the proportion of males enrolled increased the most in 
consumer and homemaking, up from 7.9 percent in 1972 to 22.6 percent in 1981. Of the 
program areas dominated by males, the greatest gain for females occurred in agriculture, up from 
5.3 percent in 1972 to 20.2 percent in 1981. Change in other areas, however, such as trade and 
industrial programs and office occupations, has been much less.^^ The fact that progress has 
been made suggests that one should not underestimate the value of rfietoric. Stating explicitly that 
one of the major objectives of federal vocational education policy is the elimination of sex 
discrimination in vocational programs has provided activists promoting gender equity with an 
important tool. 

Efforts to improve programs for handicapped and disadvantaged students appear to have 
i^et with less success. While states and local school districts have generally complied with the 
setaside provisions and matching requirements of the 1976 Amendments, there was nothing to 
prevent them from spending these funds on low quality programs. Moreover, the regulations* 
peculiar definitions of "excess costs," which arc the only expenses that may be reimbursed with 
federal setaside funds, created strong -ncentives for districts to segregate handicapped and 



^■^Charles S. Benson and E. Gareth Hoachlander, Descriptive Study of the Distribution of Federal, State, and 
Local Funds of Vocational Education: Final Report, pp. 223-234. 

^^National Center for Education Sutistirs» The Condition of Vocational Education, Washington. D.C.: U.S. 
Go''ernrru!nt Printing Office, p. 57. 



disadvantaged students in separate programs, in direct contradiction of clearly stated goals to 
"mainstream'* as many students as possible.^ 

Planning, Evaluation, and Reporting 

Although the amendments were vague on funds allocation, sex equity, and program 
improvements for students with special needs, they were much more explicit about requirements 
for planning and evaluation. To strengthen procedures for planning, evaluation, and reporting, 
the 1976 Amendments required three major documents from the state — a five year plan, an 
annual plan updating the five year plan, and an annual accountability report describing how funds 
had been used and what had been achieved. The plan4ung procedures were to give greater 
emphasis to improving the match between labor market needs and the kinds of vocational 
education programs provided. The GAO report had been very critical of the relevance of many 
vocational education programs, finding that most of the enrollment was "concentrated in 
programs with only a peripheral relationship to labor market needs.'*^^ In most states, however, 
these requirements had little effect other than to generate mountains of paper that parrotted back 
federal regulations and legislative language. 

Better planning would require better data, and the legislation addressed this need in two 
ways. First, it directed each state to establish a State Occupational Information Coordinating 
Comniittee (SOICC) and called for a national mmittee to be established as well. These 
committees were charged with improving the quality of information on labor market needs. 
Second, the act established the national Vocational Education Data System (VEDS) to collect 
comparable data from each state on vocational education enrollments, staff, expenditures, and 
student follow-up. Theoretically, data from VEDS, when matched with better data on labor 
market needs developed by the SOICCs, would greatly improve planning and the fit between 
labor market demand and supply. 

The implementation of VEDS touched off a storm of protest at the state and local level. The 
system's appetite for information was enormous, and because it sought comparability among 
states, it imposed uniform definitions of program types, student characteristics, and financial 
accounts that frequently were at odds with traditional state and local practices. The National 
Center for Education Statistics (NCES), the federal agency charged with implementing VEDS, 
endured several years of angry letters, disgruntled state director , and frustrating regional 
workshops to put VEDS securely in place. Along the way it made sc eral concessions to placate 
its critics, but it was ultimately suspended in 1983-84. 

Among vocational educators, VEDS has been the butt of many jokes, much derision, and 
some outright refusals to comply with its requests. Without a doubt, VEDS imposed a substantial 
burden on states and local school districts, and its implementation has been slow and painful. 
These difficulties, however, need to be put in perspective.The importance of VEDS can be 
understood if one recognizes how poor was the information that existed prior to VEDs. 



^^Charles S. Benson and E. Gareih Hoachlandcr, Descriptive Study of the Distribution of Federal, State, and 
Local Funds of Vocational Education: Final Report, pp. 21 7-223. 

26U.S. Consrcss, House, Cr mmittee on Education and Labor, The Vocational Education and National 
Institute of Education Amendments of 1976: Report to Accompany H.R. 12835, House Report No, 94-1085, 
94th Congress, 2nd Session, 1976, p. 16. 
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Prior to the 1976 Amendments, the vocational education data submitted to Washington by 
the states were regarded as extremely inaccurate; yet this information formed the basis for 
charting federal policy. The problem is reminiscent of a statement attributed to Sir Josiah Stamp, 
who lived from 1880 to 1941 and served as Her Majesty's Collector of Inland Revenue: 

The Government are extremely fond of amassing great quantities of 
statistics. These arc raised to the Nth degree, the cube roots arc extracted, and the 
results arc arranged into elaborate and impressive displays. What must be kept 
ever in mind, however, is that in every case, the figures are first put down by a 
village watchman, and he puts down anything he damn well pleases.^*^ 

Such was the problem with vocational education data. Counts of students enrolled in 
vocational education fluctuated wildly from year to year, with one state even reporting more 
students enrolled in secondary vocational programs than there were students enrolled in all of 
secondary education in the state. Information on finances was inadequate, and the data on the 
employment status of those who had been enrolled in vocational education was completely 
unreliable. 

As one of its first tasks for NIE, PONVER undertook an assessment of three national data 
bases that existed prior to VEDS — information collected by the Bureau of Occupational and 
Adult Education (BOAE), information produced by Project Baseline (a federally funded, five- 
year project to improve vocational education data), information collected by System 437 under 
the administration of the Office of Planning, Budgeting, and Evaluation in the U.S. Office of 
Education. PONVER submitted its report to NIE in October 1980: 

From reading the hearings and listening to the conmients of those familiar 
with vocational education data, we were expected to find problems. Despite these 
warnings, we prepared neither for the magnitude nor the pervasiveness of the 
difficulties of using these data. Our analysis led us to four general conclusions: 

(1) There is no agreement among the three data sets about either the amount 
of money expended on vocational education or the number of students served. 
For individual states, it is common to find one system reporting a figure as little 
as one-half or as much as twice that reported by another system for the same 
variable. 

(2) None of the three data sets is internally consistent over time. Annual 
fluctuations of plui or minus 30 percent in the same variables are typical and in 
almost no instance credible. . . 

(3) The two data systems administered by the U.S. Office of Education 
suffer from serious design flaws that make both very difficult to use for analytic 
purposes... 

(4) ... there is presently no national data system that permits analysis of 
how states distribute federal funds to eligible recipients. Consequently, with 



Attributed *o Sir Josiah Stamp of Her Majesty's Inland Revenue, 1880-1941. 



existing national data, it is not possible to determine how successful 
Congressional objectives have been in influencing the distribution of funds.^s 

It would be convenient to dismiss these problems as merely the result of so much variation 
among states that a uniform national reporting system is impossible to attain. Each state could be 
capable of managing its own vocational education system but unable to meet uniform national 
requirements for information. However, the facts arc that prior to VEDs, few states had any 
sophisticated management information systems of their own. Most states were not simply 
providing poor information to Washington, while relying internally on more reliable and useful 
information of another sort. Rather most states suffered from a paucity of accurate data, and 
many a state director would admit privately, if not pubhcly, that the introduction of VEDS 
provided the excuse to demand more accurate and comprehensive information for setting state 
policy. Nevertheless, lobbying to eliminate or curtail VEDS was intense. 



VEDS Outcomes 

Unfortunately, better data were not forthcoming. VEDS encountered major difficulties in 
virtually all aspects of its data collection efforts, which began in 1978-79 and continued through 
the 1982-83 school year. At the outset, it was expectcid that data quality problems would 
gradually disappear, as respondents became more and more familiar with the system. These 
hopes, however, were not realized. In 1983, cross-form and cross-year checks of VEDS data 
collected during the first three years were performed and revealed serious inconsistencies in the 
data and unrealistic annual fluctuations. Three basic problems plagued the system: 

1 . Lack of comparability among states. Data were not comparable from state to state and, 
therefore, yielded misleading national totals when aggregated. 

2. Year-to-year variablility. The data exhibited excessive variation over time, which was 
difficult £0 explain. Consequently, the VEDS data could not be used to describe trends 
over time accurately. 

3. Within state discrepancies. When VEDS data from some states were compared to state 
data from other sources, many large discrepancies were observed; these could noi be 
adequately explained 

Following these findings by NCES, the Office of Management and Budget (0MB), in 
December 1983, withdrew approval for collection VEDS data for 1983-84 and 1984-85, because 
"VEDS has substantial and continuing problems collecting data which are accurate and 
meaningful." 0MB directed that the "collection should not be undertaken at this junction and 
should remain suspended until the Department [of Education] has fully implemented the plan for 
improving VEDS." As of July 1 985, a new data collection plan had not yet been finalized.^^ 

In sum, the 1976 Amendments could count few successes, and as time for reauthorization 
drew near, there was significant pressure for complete overhaul of the Vocational Education Act. 

^^Charles S. Benson, E. Gareth Hoachlander, and Bronia Lena Johnson, An Assessment of the Reliability and 
Consistency in Reporting of Vo\.^tional Education Data available from National Data Information Systems, A 
Report Prepared for the National Institute of Education Resources, University of California at Berkeley, October 
1980, pp. 2-3. 

^^Chapter 5 discusses problems of vocational education data collection in detail. 
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In 1981, completing a five year comprehensive assessment of vocational education, the 
Vocational Education Study of the National Institute of Education reached three general 
conclusions about federal vocational education policy: 

• the Vocational Education Act of 1963, as amended, attempted to 
accomplish too much with too few resources; 

• there were sometimes mismatches between the ends of Federal policy and 
the means relied upon to realize them; arid 

• realizing the ends of Federal policy depended heavily upon State and local 
policies, practices, and rcsourccs.^^ 

In short, the study described a history of ambiguous and often contradictory regulation that 
had increasingly alienated vocational administrators and educators and generally failed to realize 
the intent of Congress. 



THE CARL D. PERKINS VOCATIONAL EDUCATION ACT OF 1984 

Reauthorization of the Vocational Education Act, which was originally scheduled for 1982, 
was delayed two years as the various factions fought over the future of national policy. The 
administration halfheartedly pushed for a block grant that would have combined vocational and 
adult education at reduced funding levels, but the proposal received little support in Congress, 
even among Republicans. The American Vocational Association lobbied hard for a bill of its own 
design that was introduced in the House (H.R 4164) in spring of 1983. Entitled the "Vocational 
Technical Education Act,*' the bill was long on rhetoric about technology, economic 
development, and industry-education partnerships, but it was very short on how its goals would 
be achieved. In fact, the bill was a block grant differing from the administration's mainly in price 
tag — about twice what the administration had in mind. A version of H.R. 4164 actually passed 
the House, but not before it was radically amended to restore the funding setasides and other 
features of current legislation. 

H.R. 4164 met a cool reception in the Senate, where both Republicans and Democrats were 
of a mind to do something new and different S. 2341 paid close attention to the major findings 
of the NIE Study, and sought to clarify and simplify federal objectives for vocational education. 
The Senate bill emphasized two major aims of federal policy: improved access and improved 
program quality. Additionally, it stressed greater private sector involvement in vocational 
education, requiring that a majority of the members of state advisory councils be from business 
and industry and establishing "industry education partnership training program in high 
technology occupations." 

The House and Senate bills were so different that most observers held little hope that a 
compromise could be struck before the October 1984 recess. However, the death of 
Congressman Carl Perkins, long a champion of federal involvement in vocational education, 
produced strong pressures to reach agreement Members in both houses saw new legislation as 
offering the opportunity to create a tribute to Perkins- Moreover, with debates over the federal 
deficit likely to dominate the 1985 session of Congress, many members and staff believed that if 
a bill could not be voted out in 1984, reauthorization would be delayed until 1986 or later. 
Consequently, after a frenzied, last-minute set of negotiations, the Conference Committee 



^%ational Institute of Education, The Vocational Education Study: The Final Report, pp. xi-xii. 
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reached agreement on October 2, 1984. The Conference Report explained hovi differences had 
been resolved on 258 poinis of conflict. 

Although the new act bears Carl Perkins' name, the strong influence of the Senate is clearly 
apparent. The legislation has two main purposes: 1) assisting the states to expand, improve, 
modernize and develop quality vocational education programs, and 2) assuring that individuals 
who arc inadequately served under vocational education programs are assured access to quality 
vocational education programs. The states arc instructed to use 43 percent their b?sic grants 
fc- purposes of program improvement, innovation, and expansion. The rcmaini'^<r 57 percent of 
the funds are to be used to fu'^d programs for students with special needs: 

• 10 percent for handicapped individuals. 

• 22 percent for disadvantaged individuals, including the ;e with limited English 
proficienc;^ , 

• 12 percent for adults in need of training or retraining, 

• 8.5 percent for homemakers and single parents, 

• 3.5 percent for participants in programs designed to eliminate sex stereotyping, and 

• 1 percent for criminal off^itders in correctional institutions. 

In addition, the law provides funds to assist states in supporting a variety of special 
programs including community-based organizations, consumer rnd homemaking education, 
adult training and retraining, career guidance and counseling, and industry-education 
partnerships in high technology occupations. It further stipulates that a majority of the members 
comprising tiit State Council shall be from the private sector. 

In several respects, the Carl Jns Vocational Education Act is an improvemc.it over past 
law. It concentrates federal fund on two major objectives that are clearly stated. It simplifies 
requirements for siatr planning and reporting, eliminating much of the repetitious parroting of 
assurances of compliance and reducing considerably the burden of collecting and reporting data 
on vocational education. Contradictory criteria for distributing funds within states have a^so been 
eliminated. The emphasis on increased private sector involvement may help to keep programs 
abreast of new developments in the world of work. 

However, if the new law is stronger on endSy it is notably weaker on means. By 
maintaining the "setaside" approach to addressing issues of access, the law provides funds for 
serving students with special needs but contains n.^ mechanism for ensuring that services and 
programs are of high quality. Furthermore, it perpetuates the unworkable notion of "excess 
costs" for determining federal support for handicapped students. Moreover, many of the 
definitions of students with special necas are so broad that, should they want to, states will be 
able to include large numbers of students under the special needs umbrella, distributing the 
federal money widely but thmly. 

Although the legislation stresses that it is the intent of Congress to concentrate federal funds 
on program improvement, innovation, or expansion rather than on maintenance of existing 
programs, what constitutes improvement, innovation, or expansion is not clearly defined. As the 
states are gi^ en complete discretion ovei how these funds will be allocated, there is a high risk 
that these funds will simply constitute general assistance, much ar they uid under the old law. 
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FinaUy. many local school districts with the greatest needs may be worse off under the new 
legislation, despite its greater emphasis on improving access. Under the old law, the 
concentration of handicapped and disadvantaged individuals affected the intrastate allocation of 
the total basic grant Under the new law, the concei xation of handicapped and disadvantaged 
students affects only the allocation of the 10 and 22 percent setasidcs. States are free to allocate 
the program improvement portion of the basic grant any way they choose. Consequently, while 
local districts with high concentrations of handicapped and disadvantaged students should wind 
up with no less money for programs for students with special needs, they may enjoy 
significantly less VEA funding overall. At the very least, the vagueness wi± which program 
improvement, innovation, and expansion have been defined will give states sufficient "wiggle 
room*' to enable them to distribute VEA funds in much the same fashion as they always have. 

In summary, although the aims of federal policy for vocational education have been more 
clearly articu ated in the new legislation, these objectives are not any more likely to be realized 
than the obje 'stives of previous law. In the next chapter we turn to possible strategies for making 
federal policy more effective and for spending federal resources more efficiently. 



CHAPTER THREE 



THE NEED FOR A NEW APPROACH TO FEDERAL POLICY 



At the risk of oversimplifying, the major problems with federal vocational education policy 
to date come down to this: 

• the federal government has sought to do too much with too little, spreading its resources 
so thinly that none of the policy goals it sets is fully and effectively realized; 

• the federal government has sought to realize its aims by prescribing for recipients of funds 
procedures to be followed rather than outcomes to be achieved. 

In FY 1984, federal expenditures of $735 million, spread over all 17 million students then 
enrolled in vocational education, amounted to less than $50 per student, or about one-tenth the 
amount that state and local gnv^:rnmcnts spent for vocational education. As such a minor 
contributor to the vocational education enterprise, the federal government could have 
accomplished its many objectives only if its goals were perfectly congruent with those of the 
states. But if such congruence existed, there would be no need for federal involvement at all. 
However, in the absence of congruence, the federal government lacks any objective measures of 
whether its aims arc being achieved. What, then, should be done? 

One option for federal policy is withdrawal from vocational education. The vocational 
ew^cation enterprise has grown enormously during the past twenty years and is now supported 
overwhelmingly by state and local dollars. The qu^ty of program offerings is uneven, but this is 
true of public education generally and may simply be the price of a highly decentralized education 
system that cannot be at once both uniformly high quality and responsive to local needs and 
desires. Similarly, while problems of access remain, a highly visible federal presence may no 
longer be required to break down barriers. Members of groups that historically have been 
underserved by vocational education are more aware than ever of their rights and have become 
increasingly effective at exercising them. In this light, federal dollars now spent for vocational 
education might be better invested elsewhere, especially if the money is simply to be left on the 
proverbial stump. 

A federal withdrawal from vocational education, however, is not likely. For one thing, the 
very success of federal policy in stimulating program growth has spawned a powerful 
constituency of vocational educators and administrators who now lobby haixl and effectively for 
federal assistance.^ While federal assistance may constitute a small portion of total spending for 
vocational education, the support has great symbolic importance to a group of educators still very 
sensitive about their status in the larger education community. They arc likely to continue to press 
for federal assistance, and Congress is likely to respond favorably. If these political 
cofiaiderations make withdrawal unlikely, how could .;deral vocational education policy be 
recast to improve its effectiveness? 



^There is no better evidence of the power and effectiveness of this lobby than the Carl Perkins Vocational 
Education Act itself. At a time when education programs and other domesuc spending were under strong attack, 
federal vocational education spending emerged virtually unscathed. 



FROM PRESCRIPTIVE TO PERMISSIVE PLANNING: THE ELEMENTS OF A 
NEW APPROACH 



For federal vocational education policy to overcome the major deficiencies that have 
plagued it in the past, the following conditions need to be met: 

• federal resources must to be concentrated on a few, clearly formulated objectives; 

• legislation and regulations must become less concerned with prescribing processes and 
insteau specify expected results and acceptable means for measuring progress toward 
these aims; 

• the distribution of federal funds must relate directly to performance on carefully specified 
outcome criteria. 

This approach to formulating federal policy has been called "permissive planning. "2 The 
distinguishing feature of permissive planning is its emphasis on outcomes rather than inputs or 
processes as criteria for evaluating program effectiveness. In determining what programs to 
expand, contract, eliminate, or initiate, the primary test is an appraisal of the programs* payoffs 
for those who participate. At the same time, permissive planning tolerates great diversity in the 
organization, funding, and deliveiy of services. What counts is that the efficiency of the choices 
be demonstrated in terms of clearly defined measures of program outcomes. 

For permissive planning to work effectively, the allocation of federal funds for vocational 
education would need to be directly related to meas);res of program performance. For example, 
local providers of vocational education could receive more federal money the greater their 
enrollments in advanced programs meeting specified performance standards and the greater their 
enrollments in these programs of students who have traditionally been underserved — the 
disadvantaged, the handicapped, students with limited English proficiency, displaced 
homemakers, single heads of households, and so forth. This practice would spread federal 
vocational education dollars over a smaller group of students than is now receiving federal 
support, increasing the incentive to offer high quality programs and to increase enrollments of 
underserved students. 

With funding tied to program performance, the federal government would be indifferent to 
how much money states and LEAs spent on handicapped and disadvantaged students. Nor 
would it be concerned wich how programi, were offered or what support services were provided. 
Rather, it would require evidence on the number of handicapped and disa^ivantaged students 
participating in vocational education programs, the nuinbcr completing programs for which there 
is a high probability of employment and good wages, the number actxially employed or pursuing 
further education or training, the numJcr whose scores rose on tests of basic or occupational 
skills, the change in performance of studenis with special needs relative to that of other students, 
and so forth. Evidence of accomplishment on such criteria would be the 'ondition for funding, 
not the ability to demonstrate that dollars were allocated in a particular fash m. 

Such an approach to allocating funds would be a radical departure from the current reliance 
on setasides, which require LEAs to use specific percentages of their funds for certain purposes 
but do not require any particular results. Consider, for example, the requirement that specific 
portions of funds allocated under the basic grant be expended on programs for handicapped and 



^Melvin M. Webber, "Planning in an Environment of Change." The Town Planning Review, Vol. 39, 
No. 4, Jan. 1969. 
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disadvantageu students. There is no requirement that states or LEAs demonstrate that 
Handicapped and disadvantaged students actually benefit from the expenditure of these funds. 
Even if the funds were used to shunt such students off into inferior programs, such practices 
would technically be in compliance with federal law. A performance-based system would prevent 
this sort of mindless compliance. 

If effectively designed and implemented, emphasizing outputs would free local vocational 
administrators and educators to do what they do best — to carry out a number of different 
approaches, carefully tailored to local circumstances and the needs of individual students. Uncier 
permissive planning, there would be no one best way, no minimum expenditure, no essential 
activity for realizing such federal objectives as program improvement or greater access. There 
would be uniform, clearly stated standards for what constitutes program success, but there 
would be any number of ways of realizing such aims. 



PROSPECTS FOR PERMISSIVE PLANNING 

Permissive planning is not a new idea. It is, in fact, the classical planning paradigm, which 
requires setting goals and priorities, understanding the influence of various phenomena on 
desired outcomes, predicting the consequences of alternative courses of action, weighing costs 
and benefits of the alternatives, and continuously monitoring outcomes to inform the resetting of 
goals and priorities. It does not free policy of regulation, evaluation, or reporting; rather it 
redirects such activities toward assessing accomplishments rather than examining compliance 
with practices or procedures related to servxe delivery. To what extent could permissive 
planning be applied under existing law? 

The Carl D. Perkins Vocational Education Act of 1984 takes an important first step toward 
practicing permissive planning; it clearly focuses federal policy on two objectives: 

• providing general support for the improvement and expansion of high quality vocational 
education programs; 

• providing additional support for improving the access of undcrserved populations to these 
same advanced programs. 

While these concerns are not new, and indeed have dominated federal vocational education policy 
for the last twenty years, the 1984 Act states them more clearly and limits federal aims to these 
two major goals. Unlike previous legislation, the 1984 Act does not attempt to deal with 
unemployment problems, economic development, equalization of resources among geographic 
regions, or a variety of other policy concerns included in past vocational education legislation. 

Although the legislation limits and articulates federal objectives clearly, it suffers from two 
major deficiencies that prohibit adopting a permissive planning approach. First, its two primary 
criteria for evaluating program performance — job placement and employer satisfaction — are 
inappropriate and unworkable in the absence of other measures. Second, funding is in no way 
dependent on performance. Rather, the legislation perpetuates the setaside approach of the 
previous act. Both of these features make it unlikely that the new lejislation will have a 
demonstrably better impact on program access and program improvement than previous policy. 



The Need for Better Performance Measures 

Federal vocational education policy has not ignored the need for performance criteria. The 
1976 Amendments to the Vocational Education Act established two primary outcome-oriented 
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criteria for program evaluation: 1) the extent to which those taking vocational education found 
employment in occupations related to their training, and 2) the extent to which their employers 
considered them to be well trained and prepared for employment. Neither of these, singly or 
together, has provided a workable standanl by which to hold program providers accountable. 

Vocational educators have objected to jc* ment and employer satisfaction as standards 
on several grounds. Pint, they have maintai ..«^t the emphasis on placement rates fails to 
distinguish vocational education from more narrow job training. Adopting placement as the 
primary criterion ignores the multiple goals of vocational education. In addition to imparting 
specific job skills, vocational education is also concerned with the acquisition of basic skills in 
reading, writing, and mathematics, and with general vocational skills that will serve students in a 
variety of ways as their careers advance. 

Second, vocational educators have also argued that the employment of vocational education 
students is determined by a large number of economic and personal factors beyond the contiol of 
the vocational education system. "Hold us accountable for employ ability^ but not for employment 
has been a frequent refrain.^ 

Third, many vocational educators have expressed concern that the single-minded focus on 
placement encourages programs to admit only those students who are likely to be easy to place. 
Consequently, important objectives — such as serving the needs of students with special 
learning disabilities and opening certain occupations to women and minorities — are ignored. 

In additi(^ to these conceptual problems, good data on placement and employer satisfaction 
have proven very expensive to obtain and unreliable even when serious efforts have been made 
to collect this information. VBDS required states to follow up students enrolled in occupationally 
specific programs six months after they had completed or left a program. Information on 
employment status was sought from students, and for those who were employed, their 
employers were surveyed to determine how well students had been prepared for work. Response 
rates were so low that the data were useless. In California, for example, at the postsecondary 
level, the status of 75 percent of those students surveyed was unknown in the 1980-81 follow- 
up. Most states and LEAs simply lacked the fmancial resources and often the technical skill to 
perform effective follow-up. 

Even had VEDS been able to achieve acceptable response rates and quality of data, the 
information would have had limited utility for evaluating the effectiveness of different vocational 
education programs. Limiting follow-up to the first six months following program completion 
does not allow enough time to capture fully the effects of vocational education. Moreover, VEDS 
collected no information on students who had taken no vocational education, so that comparisons 
of the effects of different curricula could not be evaluated. In short, obtaining good information 
on job placement and employer satisfaction is difficult and expensive to do properly. It can be 
done, but not with the annual, universal census approach adopted by VEDS."* 



^Gerry Hendrickson, Evaluating Vocational Education: The Federal Stimulus, Vocational Education Study 
Publication No. 5, Washington, D.C.: National Institute of Education, March 1981, p. 7. 

^Thc federal government has sponsored a number of longitudinal studies of small, national samples of 
secondary students, and these have produced very good follow-up information on students enrolled in vocational 
education. These studies, however, suffer from two shortcomings. They lack geographic specificity, making 
comparisons among LEAS and among states impossible, and the degree of program specificity is limited to about 
20 or mc largest vocational education programs. See Chapter 5 for a detailed discussion of vocational education 
dat? 



The Need for a More Rational Approach to Allocating Federal Funds 



To allocate federal vocational education funds to LEAs, the 1976 Amendments directed the 
states to take into account four major factors: 

• relative fmancial ability; 

• relative concentration of low-income families or individuals; 

• location in an economically depressed area; 

• intention to provide new programs. 

Although the legislation did not specify how states were to use these factors to allocate funds, 
subsequent regulations required the states to develop funds distribution formulas that not only 
included these factors but also complied with the setaside provisions for handicapped and 
disadvantaged students. As was noted in the previous chapter, the formula requirement produced 
chaos and confusion that never was resolved, and Congress understandably abandoned the 
formula requirement when it came time for reau^orization. 

Unfortunately, lacking any workable outcome measures, policy makers were forced to rely 
even more heavily on setasides to allocate federal funds. The new law divides the basic state 
grant in to two large pots; 57 percent of the money is to be used to improve vocational educadon 
opportunities for students who historically have been undcrservcd, and 43 percent is to be used 
for program improvement, innovation, and expansion. The 57 percent of the funds reserved for 
improving program access consists of 6 setasides: 

• 10 percent for handicapped individuals; 

• 22 percent for disadvantaged individuals; 

• 12 percent for adults in need of training or retraining; 

• 8.5 percent for single parents and homemakers; 

• 3.5 percent individuals participating in programs designed to eliminate sex stereotyping; 

• 1 percent for criminal offenders in correctional institutions. 

The states are directed to allocate the handicapped setaside to eligible recipients (LEAs) based on 
the relative number of economically disadvantaged individuals enrolled in each eligible recipient 
and on the basis of the relative number of handicapped students served in vocational education 
programs by each eligible recipient. The disadvantaged setaside is to be allocated on the basis of 
the relative number of economically disadvantaged individuals enrolled in each eligible recipient 
and on the basis of the relative number of disadvantaged individuals and individuals with limited 
English proficiency served in vocational education programs. The states are left more or less to 
their own devices in determining how to allocate the remaining setasides. 

Such an approach to intrastate allocation is not likely to improve access to high quality 
vocational education programs. Fully half of the funds allocated to an eligible recipient under the 
handicapped and disadvantaged setasides will be distributed simplv on the number of students 
enrolled in a recipient, regardless of whether they have access to vocational education. To take an 



extreme example, under this arrangement a district with no bandica >ped or disadvantaged 
students enrolled in vocational education could still receive setaside funds. 

The other half of the handicapped and disadvantaged setaside will be distributed based on 
the number of handicapped and disadvantaged students served by vocational education. ''Served'' 
is not defined, although it will probably be interpreted to mean the number of students enrolled in 
vocational courses or jn'ograms. Enrolhnent is a poor measure of access. In order to maximize 
the enrollment count, eligible recipients will be tempted to define vocational education as broadly 
as possible - including, for example, all students enrolled in typing courses regardless of their 
educational objectives. Further, enrollinent in vocational education generally creates nc incentives 
for recipients to increase access to more advanced higher quality components of the vocational 
education curriculum. A frequent complaint from those concerned about the access of 
handicapped students to vocational education is that handicai:^>ed students have been relegated to 
introductory courses in industrial arts and consumer and homemaking. Basing the setaside 
distribution on the number enrolled in vocational education generally will do nothing to rectify 
such practices. 

Finally, mere enrollment in a vocational education course or program is meaningless unless 
relevant skills and competencies are actually acquired A handicapped studerit enrolled on census 
day and dropping out the next day will earn a recipient just as much setaside assistance as a 
handicapped student who completes two full semesters of vocational education instruction. 
Indeed, unscrupulous recipients would be within their legal rights to load vocational courses with 
students eligible for setasides, with the expectation that many would drop out soon after the 
enrollment census day. 

Such problems are not limited to that part of the basic grant to be used for improving 
access. The distribution of the 43 percent to be used for program improvement, innovation, and 
expansion is also likely to be ineffective. The new legislation gives the states complete discretion 
as to how these funds are to be allocated to recipients. It simply lists 24 activities for which these 
funds may be expended This menu of activities is so large and many are so vaguely defmed that 
it is difficult to imagine that any recipient could not find a way to spend its relatively smaU federal 
allotment without changing a single aspect of its optiations. For example, these funds may be 
used for, among other things: 

• improving career counseling and guidance; 

• the acquisition of equipment and of facilities; 

• programs of modem industrial and agricultural arts; 

• placement services; 

• the acquisition of high-technology equipment; 

• the acquisition and operation of communications and telecommunications equipment 

These are expenditures that most recipients make routinely on an annual basis, and while they do 
not constitute a major portion of total vocational education expenditures, neither does the federal 
dollar. It will not take a very creative accountant to figure out that as long as these routine 
expenditures are financed with federal funds, the recipient will be in compliance with foleral law. 

In short, as long as the allocation of federal funds is based on spending money on 
particular types of students or on paidcular types of instructional inputs, policy is not likely to 
have a demonstrable effect on educational outcomes. Under current law it is both legal and easy 



for a recipient to spend federal setaside money on handicapped and disadvantaged students 
enrolled in mediocre vocational programs. This is not improved access, it is meaningless 
reshuffling of students around in a bankrupt curriculum. Similarly, it is both legal and easy for a 
recipient to purchase the latest and fanciest equipment for a vocational program with no regard 
for whether the equipment is appropriate for the skill level of students and faculty or likely to 
increase significantly the acquisition of basic skills and competencies needed to perform 
effectively in a job related to training. This is not program improvement, it is blind pursuit of an 
educational holy grail. Sadly, these practices occur all too often, and the failure of federal policy 
to discourage them undermines those who do make careful and thoughtful efforts at real 
improvements in access and vocational education programs. 

It can be argued, of course, that the educational system is inherently resistant to planning, 
evaluation, and resource allocation based on outcome criteria. For example, Hendrickson notes: 

Decisions on whether to continue a program are influenced by community 
pressure, stuoent demand, presence of a tenured teacher, and, to a lesser degree, 
data. In fact, at the secondary level, student demand is likely to be the primary 
determinant of program offerings. In calling for student placement data and in 
encouraging their use in program planning. Congress is superimposing a rational 
process on a political one.^ 

Successful politics, however, docs ultimately hinge on performance, and despite the large body 
of experience that would seem to contradict it, politics and rationality can co-exisL Indeed, as we 
have argued here, reliance on student placement data as the primary criterion for vocational 
education program planning evalution is not only conceptually flawed but also, given the quality 
of the data, subject to serious error. The decision to ignore the data, therefore, may be political, 
but it is also rational. The decision docs not imply that the planning and evaluation process need 
be inherently input-oriented but rather that the specification of performance measures must be 
done more carefully and thoughtfully. Moving from a prescriptive to a permissive planning 
mode, therefore, ultimately depends on how well program outcomes can be defined and 
measured, subjects we take up in detail in the two chapters that follow. 

Federal policy is not without some experience with permissive planning. The Job Training 
Partnership Act (JTPA), enacted in 1982, adopted a performance-based approach to the 
evaluation and funding of federally assisted job training programs, and a brief review of JTPA is 
a useful preamble to considering how permissive planning might work in vocational education. 



LESSONS FROM JTPA 

The Job Training Partnership Act (JTPA) of 1982 (P.L. 97-300) signaled a new 
development in federal policy concerning job training. JTPA eschews many of the prescriptive 
features of the Comprehensive Employment and Training Act (CETA) that preceded it. JTPA is 
relatively silent on what types of p'^ograms must be offered, who must offer them, and how this 
must be donr Rather, it urges careful specification of what outcomes training programs are 
expected to achieve and requires that service providers be held accountable for attaining these 
ends. In short, it prescribes results, while permitting a practically limitless variety of procedures 
and processes for obtaining these objectives. 



^Gerry Hendrickson, Evaluaung Vocational Education: The Federal Stimulus, p. 30, footnote omitted. 
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JTPA adopts some basic tenets of human capital theory and 'applies principles of 
cost/benefit analysis to assess the rate of return for various training programs. It adopts as the 
basic measure of program performance "the increase in employment and earnings and the 
reductions in welfare dependency resulting from participation in the programs."^ Programs 
whose costs exceed benefits arc to be dropped, while those demonstrating positive rates of return 
are to be continued or expanded. 

The Department of Labor set performance standards requiring youth training programs 
(serving disadvantaged youths aged 16 to 21 years) to have job placement rates of at least 41 
percent and "positive-termination" (employed, in school, or in the armed forces) rates of 82 
percent Placement, however, was not the sole criterion for evaluation. The local Private Industry 
Councils (PJCs) were also directed to develop sets of "competencies" that program participants 
would be expected to acquire. PICs could count as positive terminations participants who were 
placed in jobs or who successfully mastered the specified competencies. PICs successfully 
achieving their positive termination goals are rewarded, and those failing to do so are required to 
reorganize their training programs. 

PTA has been in operation for just over two years, and there is as yet only limited 
information on its effectiveness. Initial studies have shown that, overall, job placement rates h: ve 
been high, with many sites equaling or surpassing their placement goals for all categories of 
enrollees with one important exception, disadvantaged youths. The failure to train disadvantaged 
youths as effectively is troublesome. Not only arc they one of the primary targets of JTPA, but 
also the difficulties of meeting performance goals with this target group may increase the 
tendency of trainers to "cream, " admitting only those individuals who are most likely to be 
placed in jobs and ignoring those hardest to serve. The evidence on whether creaming has 
actually occurred in JTPA programs is mixed* Some studies purport to have found it; others have 
not. Most report that program administrators complained that the focus on high performance 
standards diminishes their willingness to take risks, either with individuals or with innovative 
training programs.^ 

Although these findings are tentative, representing only very early experience with JTPA, 
they underscore the fact that adopting permissive planning in vocational education will work only 
if the performance sf indards are carefully designed to encourage the kinds of behavior desired. 
This is the task of the next chapter. 



6p.L. 97-300, Sec. 106 (b)(1). 
^Education Week, January 23, 1985, p. 1. 
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CHAPTER FOUR 



AN APPROACH TO PERFORMANCE-BASED EVALUATION 
OF VOCATIONAL EDUCATION PROGRAMS 



At the federal and state levels, planning and evaluation of vocational education programs 
have traditionally been process-oriented. These efforts have been mainly concerned with such 
issues as the distribution and expenditure of different sources of funds, the types of curricula and 
special services provided, the size and composition of various advisory committees, the extent ♦o 
which local labor market data are used to expand and contract programs, and so forth. 
Permissive planning, while it is not completely indifferent to these process issues, places more 
emphasis on determining what is accomplished than on analyzing how it is done. Permissive 
planning, therefore, requires a performance-based approach to evaluating and planning. 
Developing such an approach is the primary goal of this chapter. 

The performance-based evaluation system developed in this chapter is intended primarily 
for assessing vocational education programs that have as one of their primary :>bjectives 
preparing stuaents for entry level employment in occupations that do not require a baccalaureate 
degree. Consequently, it is aimed at occupationally speciHc programs, which typically are 
offered in grades 1 1 and above, including postsecondary. It is not intended as a f^uitable method 
for directly assessing introductory programs or programs in industrial, arts or consumer and 
homemaking education.^ Furthermore, it is intended to improve the allocation of federal aid to 
vocational education, and perhaps in some cases state categorical assistance, at the margin. We 
take as given, therefore, that federal aid is not intended for general program support but rather to 
encourage and reward program improvement The performance-based evaluation system is not 
intended as a primary tool for allocating general state and local funds for vocational education.^ 

An effective system of performance-based evaluation must meet three important criteria. 
First, it must use appropriate measures of program effectiveness. These must be quantifiable and 
well defmed. Second, the approach must be adaptable to quantitative methods and procedures 
that states and LEAs can follow to evaluate program effectiveness. These procedures should 
permit comparisons across programs and LEAs. Tliird, it must be possible to tie the approach to 
a set of fiscal incentives that encourage and reward program improvement. If program 
performance has no bearing on financing, there is no reason for service providers to take 
evaluation seriously, and compliance concerns will continue to focus on inputs and procedures 
rather than program accomplishments. Successful implementation of the approach will also 
require accurate and timely data, but this subject will be treated separately in Chapter Five. 



^The results of the program evaluations could be used to flag certain introductory programs for further 
examination. Most introductory programs feed occupationally specific programs, and if a particular occupationMly 
specific program performs poorly, assessing the introductory program that precedes it may help to improve future 
oerformance. 

^The results of these program evaluations could be used to evaluate the allocation of general aid, particularly 
whether certain programs ought to be continued. However, the system developed here, while providing a method 
for distributing categorical aid to vocational education, does not constitute a full-blown allocation system for all 
vocational education funding. 
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THE OUTCOMES OF VOCATIONAL EDUCATION PROGRAMS 



Historically, the primary goal of vocational education has been to prepare students for 
employment in occupations requiring less than a baccalaureate degree. Curiously, although the 
major concern of vocational education is preparation for employment, tb<* primary test of the 
quality of vocational education programs has been employment itself, i.e., placement in a job 
related to training. There are many reasons why placement constitutes a poor measure of the 
quality of preparation. First, a program's placement rate is heavily influenced by general 
economic conditions. If demand for labor is strong in a particular field, even programs producing 
pooriy prepared graduates may enjoy high placement rates; conversely, if demand for labor is 
weak, graduates of even the finest programs will have difficulty finding jobs. Placement, 
therefore, may often be more an indication of the demand for labor in a field than an indication of 
the quality of particular vocational education programs.^ 

Second, the emphasis on placement in afield related to training may often be misguided. At 
its best, vocational education programs provide students with a generic set of academic and 
work-related skills that will enable them to better adapt to a variety of occupational demands and 
to change occupations more easily when economic conditions or personal choice dictate it. 
Vocational education ought also to facilitate the pursuit of further education or training. Thus, 
particularly with less advanced programs, mere employment (regardless of the field) or 
continuing education may be as good or better indicators of program effectiveness than 
employment in a related field As programs become more advanced or specialized, however, the 
related field requirement assumes greater weight, because the specialized training is not easily 
transferred to other occupations and is therefore wasted if the graduate chooses or, because of 
economic conditions is forced into, another field. 

Third, just as placement is heavily influenced by economic conditions, ic is also affected by 
discrimination and various types of stereotyping that limit the employment prospects of certain 
types of students regardless of how well prepared they may be. Vocational education has as one 
of its highest priorities improving the access of students with special needs — minorities, men 
and women wishing to pursue occupations not traditional for their sex, the handicapped, the 
disadvantaged etc. — to the highest quality of program consistent with their abilities. However, 
even if ycficational education programs adequately prepare these students to perform in a particular 
occupation, they will face major obstacles in the labor market that diminish their chances for 
employment In these instances, placement rates measure not so much the quality of the training 
program but rather the extent to which various kinds of discrimination continue to affect hiring 
decisions. 

Fourth, even if placement were an appropriate measure of a programs's effectiveness (and 
under carefully controlled conditions, we think it can be one of several useful indicators), 
accurate and useful placement data have proven exceedingly difficult and expensive to obtain. 
Response rates for VEDS follow-up efforts were unacceptably low. In California, for example, 
despite the substantial investment of time and money at the secondary level in the FoUow-Up of 
Students and Employers (FUSE), the employment status of 31 percent of the secondary students 
was unknown in the 1981 data collection. At the postsecondary level, the follow-up data were 
completely useless, with the employment status of over 70 percent of the students unknown. 
Information reported by employers was even less complete. Unfortunately, these follow-up 
results from California were typical of most states conducting the VEDS data collection. 

demand for labor in a particular field is weak, it can be argued that a program, no matter how exemplary, 
ought not to be offered at all. If weak demand is expected to persist for an extended period of time, this is a 
reasonable position. Note, however, that under conditions of protracted low demand, it is the quality of overall 
progtzm planning, not program content or the quality of preparation, that is at issue. 
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In short, despite the popularity of placement data with Congress and many state 
legislatures, placement is a poor measure of how effectively programs prepare students for 
employment, especially if it is applied indiscriminantly as the single measure of program 
outcomes. What, then, constitute some more appropriate alternatives? We recommend 
concentrating on four types of outcomes: 

• Employability 

• Acquisition of generic academic and work-related skills 

• Access 

• Post-completion status 

By "employability," we mean the acquisition of basic and job-specific skills required to 
perform effectively in entry level jobs related to training. We will have more to say below on 
how these skills are identified and how appropriate levels of performance are established. For 
now, let us simply assume that relevant skUl levels can be determined for every entry level job in 
occupations not requiring a baccalaureate degree. Then, through a variety of competency testing 
procedures, the acquisition of these skills can be assessed quite independently of actual 
employment. An effective program, therefore, is one that successfully imparts these skills, 
making a student employable in a particular field. Employability, we believe, should be the 
primary criterion for evaluating the effectiveness of particular programs. Placement and other 
employment related measures are relevant for assessing the effectiveness of program planning, 
and secondarily the thorough and accurate specification of basic and job-specific skills required 
to perform a particular job, but they are not well suited to assessing how well students are 
prepared for work. 

Employability represents sufficient preparation to perform a specific job. We consider it to 
be the most important program outcome and would not consider any student to have completed a 
vocational education program unless he or she has become employable. Indeed, in the approach 
to program evaluation we arc proposing, program completion is synonomous with employability 
and certifies that the completing student has acquired the basic and job-specific skills necessary to 
perform effectively in an entry level job related to training. This definition has important 
implications for the procedures proposed for distributing federal funds, as well as data collection 
requirements, topics that will be addressed in more detail below. 

Although program completion, or employability, constitutes the most desirable outcome to 
be achieved by a vocational education program, the acquisition o^ ; wicral academic and work- 
related skills is also desirable, even though these skills arc not sufncient to ensure employability. 
They are transferable to other kinds of further academic or job-specific training and will 
accelerate that training if previously acquired. Therr^c;e, we consider the acquisition of general 
academic and work-related skills to be another important program outcome. 

Access cannot be assessed independently of employability or general skills, because if 
participation in a vocational education program does not, at the very least, kad to the acquisition 
of general academic and work-related skills or, more desirably, to employability, access is 
meaningless. A program that enrolls students with special needs but neither makes them 
employable nor increases their general skills perpetrates a cruel hoax. Enrollment is a necessary 
but not a sufficient condition for ensuring that special needs students benefit from the program, 
and the approach proposed here seeks to reward only those programs that can demonstrate 
results, preferably by increasing the number of special needs students completing the program 
or, at the very least, by demonstrating the acquisiuon of some general academic and work-related 
skills. 
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Finally, while we would still include post-completion follow-up outcomes in assessments 
of how well students arc prepared for work or further education, we consider these unsuited to 
large scale, ongoing evaluation and would examine them only when the data are accurate and 
complete enoujgh to permit using sophisticated statistical techniques to control for the large 
number of variables other than program quality that influence employment outcomes (e.g., 
economic conditions, hiring practices, student characteristics, etc.). The appendix to this chapter 
develops a statistical model for examining the relative effectiveness of different vocational 
education programs with respect to employment outcomes. We would employ it periodically 
(approximately every r;vo to four years) to evaluate how well standards of completion have been 
specified for different vocational education programs and to assess vocational education generally 
with respect to other parts of the secondary and postsecondaiy curriculum. 

For purposes of federal and state policy, ongoing program evaluation should concentrate 
mainly on the first and third of these performance criteria, employability and access. Both may be 
uniformly defined, and neither imposes an uni'easonable burden in collecting and reporting 
accurate information. While monitoring general skills acquisition is desirable at ^e local level, it 
is more difficult to define in uniform, and therefore comparable, terms and would impose a 
substantially greater data burden. Moreover, whereas by definition general skill acquisition is a 
precondition for employability, wliich requires some job specific s^Us in addition, monitoring 
the endpoint in the preparation process will provide some good indications of how well the 
intermediate steps are being performed Examination of follow-up data at the state or federal level 
should be limited to infrequent but in-depth analyses of sample data to assess and validate the 
ongoing evaluations of employability and access. 

SOME OPERATIONAL DEFINITIONS FOR STATE AND FEDERAL EVALUATION 

For purposes of state and federal evaluation of vocational education programs, three types 
of program outcomes must be defined: employability, access, and post-completion status. We 
shall take these up in turn. 

Employability 

In the system proposed here, employability is measured by program completion, which is 
defined as "satisfying the requirements for a degree, certificate, diploma, or other formal award 
and completing a vocational education program that certifies that the student has acquired the 
general and job-specific skills necessary to perform effectively an entry level job in an occupation 
related to the student's training." At the sccopdary level, therefore, students could be counted as 
completers of vocational education programs only if they also earn a diploma.^ We recommend 
linking completion to some form of graduation requirement for two reasons First, in today's 
economy the long term employment prospects of a student failing to finish high school are dim 
indeed and are not likely to be helped much by a vocational education program alone. It is highly 
desirable for vocational education programs to have powerful incentives to encourage their 
students to finish high school. Second, the requirement greatly simplifies data collection and 
reporting. A vocational education student need be counted only once in his or her educational 



'^Because many handicapped students are incapable of fulfilling the requirements for a high school diploma, the 
definition of a program completer needs to be modified son^what for handicapped students in vocational education. 
For handicapped students, program completion should be defmed as ''satisfying the requirements for a diploma cr 
completing the requirements of an lEP, or reaching the age of 21, and completing a vocational education program 
that certifies that the student has acquired the general and job-specific skills necessary to perform effectively an 
entry level job in an occupation related to training.'' 
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car er — at the time of graduation at the secondary leve) or, at the nor,tsecondary level, at the 
time of an awaid of a degree, certificate, or other formal aw ard 



Note that this definition of completion is uniform in a generic sense — i.e., everywhere it 
represents certification that the student has been adequately prepared to perform an entry level job 
in an occupation related to training. It is not uniform, however, in the sens^ that all completers of 
a particular type of program have uniformly acquired the same skills. What constitutes adequate 
preparation can and should differ depending on loca! circumstances. For example, adequately 
preparing a student for an automotive job in a rural area typically requires stressing a broad range 
of general job skills, using less costly and less specialized diagnostic equipment t'lan would be 
available in an urban area. Moreover, entry level positions in automotive occupations in 
metropolitan areas tend to be ir.ore specialized; for example, in an urban area, shops can 
special' re in muffler *ieplacement or transmission repa^/ and hire accordingly. I" rural areas, there 
^ri-^ ^ A sufficient numbers to wairant this kind of specialization ai.J an emploj must be able to 
per .^rm a greater variety of more general tasks to be fully occupied throughout the work day. 
A&.qvzit preparation for automotive occupations, therefore, requires different kinds of 
^, reparation in different types of labor market':. 

It is important that the kind of definitional uniformity required for vocational education 
policy purposes be well unr^er^st^, for there is often a tendency, esp^'cially at the federal level, 
to press mindlessly for uniformity because of the mistaken impressic that uniform definitions 
always improve data. If what must be uniformly reported is poorly thought through, the 
consequences for policy can be disastrous. For example, consider the wisdom of defining 
completion of an automotive program in terms a uniform set of skills that all students nationwide 
must acquire u crder to be considered a completer. One would be confident that when adding 
program completers in ^xral areas to program completers in urban areas, one was in fact adding 
"the same thing," but 1*^*: numbers would have no meaning for policy purposes. Th: policy 
concern is whether students have been adequately prepared to obtain entry level jobs, and if ^he 
content of these jobs differs locally, the kinds of skills imparted must diffe/ 1:) ensure adequate 
preparation. To insist on acquisition of a set of uniform skills is to ensure that all students are 
inadequately prepared, except for those able to find employment In areas with the "average" 
conditions represented by the set of uriform skills. 

The definition of program completion proposed here permits a great deal of state and local 
discretion in determining what constitutes adequate preparation. While this is certain to make 
collectors and users of state and national data nervous, it has a much greater likelitiood of 
producing useful data for policy than an attempt to impose defii 'tions w:^h specifying uniform 
skiili.^ For this approach to work, however, it must include some rigorous methods for 
validating loca'ly determined standards for program completion and modifying those stanoards 
which are foui 1 to be inconsistent with effective job performance. We shall return to this need 
below. 

^Generally speaking, vocational Cviucjltion dau have been so bad for so long (See Chapter Five) that those 
respohaioie foi ^heir collection ^r.d re^ntng have become especially sensitive to problems of data comparability 
and consistency. Unfortunately, this tends to produce an unnecessarily low tolerance for ambiguity or variability ir 
collecting some vocational education data. We requi e relatively litde "compaiability" from most of the other 
national education dau. For example, we routinely rely on reports of the national totals of students with high 
school diplomas or college degrees, knowing full well that there is substantial variation in precisely what the 
diplomas or degrees represent in erms of academic preparation and skill. Similarly, we often And it useful to knew 
how many 12ih graders are taking "math" or "scier^i,." without worrying about the large qualiutwc differences 
that exist ar .ong schools LEAs, and states. It is not at^oi why it is necessary to subject vocational education data 
collection to a stiffer standard, especially if tiie insistence on comparability means that data cannot be collected at 
all. 
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Access 



Traditionally, efforts to evaluate program acce is in vocational education have concentrated 
on monitoring the numbers of different types of students enrolled in particular vocational 
education programs relative to their numbers in the larger secondary or postsecondary 
population. This approach suffers from both conceptual and methodological flaws. 
Conceptually, the focus on enrollment provides no information on performance, cither by the 
student or the program. Unless students complete a program or unless there is hard evidence that 
prior to completion they acquired some useful academic and other work-related skills, students 
enrolled in a program arc simply marking time. Methodologically, accurate counts of program 
enrollment have proven very difficult to obtain (Sec Chapter Five). 

We reconmicnd, therefore, that the evaluation of access concentrate on monitoring program 
completions — by race and ethnicity, by sex, and by special need. With data on their 
distribution in the larger population, access by a particular type of student can be monitored using 
an "access ratio** defined as follows: 

the percentage of students of a particular r/pe completing a particular vocational 
education program divided by the percent age of students of this type enrolled in the 
laiger population. 

The relevant "lari;er population" depends on the level of aggregation at which program 
completion is being assessed — school, LEA, state, region, or nation as a whole. 

TTiis access ratio provides a simple means for making quick comparisons of access across 
different programs, schools, LEAs, and states. When a vocational education program enrolls 
students of a particular type in direct proportion to their numbers in the larger population, this 
ratio will equal 1.00. When a particular type of student is under represented in a vocational 
education program, this ratio will be less than 1.00. It will be greater fiian 1.0 when a particular 
type of student is over represented For example, if in a hypothetical comprehensive high school, 
boys constitute 25 percent of the completers of a secretarial program and 48 percent of larger 
school population, the access ratio of the secretarial program, with respect to boys, is .52 
(25/48). If in this same school girls constitute 10 percent of the completers in a masonry program 
and 52 percent of the larger population, the access ratio of the masonry program, with respect to 
girls, is .19 (10/52). As the boys* access ratio of the secretarial program, .52, is higher than the 
gLL* access n:tio in the masonry program, the secretarial program is more effective at eliminating 
sex stereotyping than the masonry program. Similar ratios could be calculated v/ith respect to 
race nJ ethnicity and various types of special need, as well as for the complete array of 
vocational education programs. 

In tho example just presented, the reader may question the need for the ratio, as it is quite 
clear from the simple percentages of boy and girl completers that sex stereotyping is more severe 
in the masonry program. We know this, however, only because we know that generally the 
ZT population is evenly divided between boys and girls and this distribution does rot vary 
significantly by school or school district Witti respect to other types of students, however, the 
simple percentage is not "Sufficient to assess access. For example, we can say nothing about the 
accessibility of a particular program to disadvantaged students if we are simply told that 5 percent 
of the completers were disadvantaged. We need to know the percentage of disadvantaged 
students in the larger population, which will vary by school, LEA, ana even by state. Tnus, the 
access ratio automaticaUy corrects for differences among schools, LEAs, and states in the relative 
distribution of different types of students in the larger population. 



Whatever the unit of analysis — schcK>l, LEA, state, or nation as a whole — the calculated 
access ratios may displayed in a form similar to Table 2. Programs in which ratios differ 
significantly from 1 OO may be flagged for further examination. 

These access ratios must be interpreted carefully, especially those measuring the access of 
handicapped students. In the case of some handicapping conditions, the handicap inherently 
limits or prevents participation in a particular program. It is not likely, for example, that one 
would find students with severe visual impairment completing an Airplane Piloting and 
Navigation program. The access ratio for visually impaired students in this program is likely to 
be 0.00; yet few would suggest that such a result is indicative of unfair denial of access. The 
problem, of course, is that what constitutes an inherent limitation and what constitutes unfounded 
assumptions about the capabilities of handicapped students or the abilities of programs to adapt to 
serve handicapped students is usually not so clear cut A ratio significantly different (either larger 
or smaller) fiom 1.00, therefore, may be regarded as indicative of possible access problems for 
handicapped students; however, it should never be used as the last word. 

Post-Completion Status 

The proposed system would use conventional measures of post-program completion 
outcomes — e.g., labor market status, educational status, type of job, wages, duration of 
employment or unemployment, etc. These arc well Wiown, and we will not belabor them here. 
How follow-up is done would differ considerably rrom past practices, however. In the past, 
federal requirements for follow-up called for surveying annually the universe of program 
completers six months after completion. Not onl; did the size of this data coUectior effort make it 
impossible to obtain accurate data, but also the ame between completion and follow-up was too 
short to produce much useful information, even if it could have been collected accurately. 
Furthermore, the follow-up effort was Imiitcd to vocational education students, preventing 
comparisons with other types of student** 

Follow-up activity should continue, but less frequently and in a fashion that follows small 
samples of students for longer periods of tL-ne and collects more extensive data. Ideally, state- 
level efforts should piggyback onto ongoing follow-up activities at the federal level. The U.S. 
Department of Education presently supports two major. longitudinal studies, the National 
Longitudinal Study of the Senior Qass of 1972 and High School and Beyond (following a 
cohort of 1980 sophomores and 1980 seniors), that provide a rich database for analyzing what 
students do following participafion in vocational education programs. A third study, the National 
Education Longitudinal Study is slated to begin in 1988, and efforts are now underway to tailor it 
for more comprehensive collection of data on vocational education. For each of these studies, the 
initial sample of schools and students is large enough to develop state-specific information for the 
eight largest states, and other states arc encouraged to augment the sample to obtain state-specific 
information that they can use for their own policy puiposes. 

State follow-up linked to the national longitudinal studies would serve two important 
purposes, jhirst, it would provide states with accurate extensive historical data for assessing their 
vocational (and non-vocational) education system. These surveys are sufficiently detailed to 
permit statistical control of the many variables, besides program characteristics, that affect 
employment and education outcomes after participating in a vocational education program. 
Second, analyses of inese data would aid periodic assessments of how well local schools and 
school districts specify the requirements for program completion. 

In summary, then, the system of progra.ii evaluation developed here would, at the state and 
federal level, concentrate on examining the number of completers produced by vocational 
education programs and rates of program completion among students by race, sex, and spec'a* 
need. States would also conduct analyses of data produced from the national longitudinal 
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TABLE 2 
ACCESS RATIO DISPLAY 





SEX 


RACE/ETHNICITY 


DISADVANTAGE 


HAND 


ICAPPIN 


G COND 


moN 






ri\v/iji\/f in 




Female 


While, D0( 


Black, not 

liispanic 


Other 
MinonlY 


Academic. 
Disadvant. 


Ecodooic. 


MestaUy 


Emo(ioiially 


Leaning 


Had of 


Speech 


Visually 
Handicaivod 


Oithopodic. 
Handicappoc 


Otiicr Health 


AgncuHurai Mahanics 
































AgncuUurai Produclioo. 
































Agncultura! ProducU 

































AgncuKuraJ Business 
































Hoiticulture 
































AgncuUure,0(ha 

































Accounting, Bookkeeping 

































DaU Proccsing 
































SecrOanal & Related 
































TvwnB. General Office 

































Bus less A OITice, QOier 
































Business & Personal Serv 
































Food Marketing 
































General Marketing 
































Hospitality & Rccrcalioo 
































Marketing ^: Dist., Other 
































Nursing Related 
































Alhcd Health, Other 
































Numng 
































Consumer Sl Homcmakini 
































Child Care & Guidance 
































Clothing, Apparel 

































5q 



surveys of small smaplcs of secondary and pofisecondary students. What would such a system 
require at the local level? 



IMPLICATIONS AT THE LOCAL LEVEL 

For a completion-based evaluation approach to work at the state and federal level, local 
procedures for certifying that completers are employable must be as rigorous and thorough as 
possible. We recommend, therefore, that competency-based curricula and competency testing 
become mandatory featiores of all vocational education programs receiving federal funds. Cuch a 
requirement is consistent with a new provision of the Carl Perkins Act, Section 113(b)(9)(a), 
which states that each state's plan for vocational education shall: 



provide assurances that the State will develop measures for the effectiveness of 
programs assisted under this Act in meeting the needs identified in the State plan, 
including evaluative measures such as: 

(i) the occupations to be trained for, which v. ill reflect a realistic assessment of the 
labor market needs of the State; 

(ii) the levels of skills to be achieved in particular occupations, which will reflect the 
hiring needs of employers; and 

(iii) the basic employment competencies to be used in performance outcomes, which 
will reflect the hiring needs of employers. 



This requirement is also consistent with the approach adopted in the Job Training 
Partnership Act (JTPA), which directs each local Private Industry Council (PIC) to develop sets 
of competencies for its youth training programs. Programs are to be responsible for developing 
competencies in at least one of four major areas: 

Pre-Employment Competencies: basic awareness of the world of work, including 
familiarity with a variety of career options, the level of education required to pursue 
each, and the likely income that can be expected from each; an understanding of 
one's own preferences, 'nterests, and aptitudes; basic survival skills, including how 
to open a checking and savings account, rent an apartment, obtain a social security 
card, make knowledgeable purchases of basic consumer items, and so forth; job 
search skills, including preparing a resume, knowing where to look for job 
opportunities, filling out an application, and being interviewed. 

Work Maturity Competencies: demonstrated abilities to meet employers' 
expectations of basic responsibilities, such as regular and punctual attendance, 
proper dress, ability to carry out instructions, ability to work with others, and so 
forth. 

Basic Education Competencies: skills in reading, writing, computing, and 
communicating needed to function successfully in the workplace, with an emphasis 
on demonstratea ability to apply these skills in real work situations. 

Job S, ""cific Competencies: basic and advanced skills required to perform 
effectively in a chosen occupation or cluster of occupations. 
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Each program must develop a set of specific, measurable competency statements along with 
appropriate "benchmarks" that establish an acceptable level of performance for each competency. 
Successful completion of the program, therefore, depends entirely on reaching the benchmarks 
that have been established for it, and the effectiveness of a program can be assessed by the 
number of successful completions it achieves. 

Competency-based instruction would not be new to vocational education. Indeed, it is 
commonplace in many area vocational schools. Additionally, several state?, such as Florida and 
New York, have begun efforts to require competency-based instruction throughout the statewide 
vocational edr.ation system. Competency-based instruction, howe^^r, is far from universal, 
absent most noticeably from most comprehensive high schools, where the majority of secondary 
vocational education still occurs. The use of competency-based instruction is also spotty at the 
postsecondary level. 

Competency-based instruction in vocational education requires careful identification and 
measurement of two general types of skills, basic skills and job specific skills. Each of these 
types can be illustrated in detail. 

BASIC SKILLS 

What kinds of basic skills are required to perform effectively on the job? A panel of 
distinguished educators and business people recently considered this question for the National 
Academy of Sciences. They concluded that: 

the need for adaptability and lifelong learning dictates a set of core competencies 
that arc critical to successful careers of high school graduates. These 
competencies include the ability to read, write, reason, and compute; an 
understanding of American social and economic life; a knowledge of the basic 
principles of the physical and biological sciences; experience with cooperation and 
conflict resolution in groups; and possession of attitudes and personal habits that 
make for a dependable, responsible, adaptable, and informed worker and citizen. 
Together these competencies comprise what are needed to prepare a young person 
for an uncertain future.^ 

ITie panel went on to list a number of skills required by employers in each of the major 
competency areas. These are summarized below to illustrate the kinds of basic skills that should 
become part of a performance-based model of vocational education.'^ 

Command of English: a functional command of standard English in its written and spoken 
forms. 

Reasoning and Problemsolving: 

• Identify problems 

• Consider and evaluate possible, alternative solutions 

• Formulate and reach decisions logicaUy 



^Panel on iecondary School Education for the Changing Workplace, High School and the Chaiging Workplace: 
The Employers' View, Washington, D.C.: National Academy of Sciences, 1984, p. 19. 

^Ibid., pp. 20 - 26. 
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• Separate fact from opinion 

• Adjust to unanticipated situations by applying established rules and facts 

• Work out new ways of handling recurring problems 

• Determine what is needed to accomplish work assignments 

Reading: 

• Understand the purpose of written material 

• Note details and facts 

• Identify and summarise principal and subsidiary ideas 

• Be aware of inconsistency in written maicrial 

• Verify information and evaluate the worth and objectivity of sources 

• Interpret quantitative information 

Writing: 

• Gather information suitable for the purpose 

• Organize information in a logical and coherent manner 

• Use standard English syntax 

• Apply the rules of correct spelling, punctuation, and capitalization 

• Attribute references correctly 

• Use reference books such as a dictionary, a thesaurus, and an encyclopedia 

• Write legibly 

Computation: 

• Add, subtract, multiply, and divide whole numbers, decimals, and fractions accurately 

• Calculate distance, weight, area, volume, and time 

• Convert from one measurement system to another, for example, from English to metric 

• Determine the costs, time, or resources necessary for a task 

• Calculate simple interest 

• Compute costs and make change 

• Understand simple probability and statistics 

• Calculate using information obtained from charts, graphs, and tables 

• Use ratios, proportions, percentages, and algebraic equations with a single unknown 

• Estimate results and judge their accuracy 

Oral Communication: 

• Communicate in standard English 

• Understand the intent and details of oral communications 

• UndCTSta ' and give instructions 

• Identify ar.a sunmiarize correctly principal and subsidiary ideas in discussions 

• Obtain, clarify, and verify information through questioning 
^ Paiticipate effectively in discussions 

Interpersonal Relationships 

• Interact in a sociaUy appropriate manner 

• Demonstrate respect for the opinions, customs, and individual differences of others 

• Appreciate the importance and value of humor 

• Offer and accept criticism constructively 

• Handle conflict maturely 

• Participate in reaching group decisions 
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Social and Economic Studies: 



• The history of present-day American society 

• The political, economic, and social systems of the United States and other countries 

• The fundamentals of economics, including a basic understanding of the roles of money, 
capital investment, product pricing, cost, profit, and productivity, and market forces such 
as supply and demand 

• The concept of "trade-offs" and the differences between economic principles, facts, and 
value judgments 

• The roles of industry and labor in creating wealth, maintaining employment, and raising 
the standard of living 

• Basic awareness of the world of work, including familiarity with a variety of career 
options, the level of education rrxiuired to pursue each, and the likely income that can be 
expected Irom each 

• The rights and responsibilities of citizens 

• Civil rights and justice in a free society 

Personal Work Habits and Attitudes: 

• A realistic positive attitude toward one's self 

• A positive attitude toward work and pride in accomplishment 

• A willingness to learn 

• Self-discipline, including regular and punctual attendance and dependability 

• The ability to set goals and allocate time to achieve them 

• The Capacity to accept responsibility and carry out instructions 

• The ability to work without supervision 

• Appropriate dress and grooming 

• An understanding of the need for organization, supervision, rules, policies, and 
procedures 

• Freedom from substance abuse 

• Appropriate personal hygiene 

• The abUity to search for a job, including preparing a resume, knowing where to look for 
job opportunities, filling oui an application, and being interviewed 

Although it is unlikely that a particular vocational education program could significantly 
improve all of these "core" competencies, concentration on different subsets by different types of 
vocational programs would be highly desirable. Automotive programs, for example might be 
particularly well suited for imparting certain mathematical concepts such as measurement and use 
of ratios. Office programs might place more emphasis on writing and oral communication skills. 
Vocational programs engaged in the production of real goods and services (e.g., a building 
trades program that builds and sells a house) would provide excellent opportunities for teaching 
the fundamentals of economics. In short, programs should be encouraged to identify those basic 
skills they are especially effective in teaching, and students' mastery of those skills should be 
tested and evaluated. 

JOB SPECIFIC SKILLS 

Job specific skills include the knowledge and skills normally required to carry out entry- 
level tasks in a specific occupation or cluster of occupations. Imparting job specific skills is the 
mission traditionally associated with vocational education. Although there is likely to be some 
disagreement over what precisely constitutes "entry level," identifying job specific skills for a 



particular occupation is relatively straightforward. An entry-level automechanic, for example, 
nriight be expected to have mastered the following specific skills: 

• familiarity with shop safety and basic shop procedures 

• knowledge of basic shop tools and equipment 

• ability to perform basic lubrication, battery, and cooling service 

• ability to perform basic tire service 

• ability to perform a basic tune-up 

• ability to service emission control equipment 

• ability to perform headlight adjustment and lamp replacement 

• ability to perform basic brake service 

• ability to perform basic front end service 

Program completion, then, should certify that a student has mastered the appropriate uux 
of basic and job specific skills required to perform entry level work effectively in an occupation 
related to training. For example. Table 3 lists the basic and job specific skills required by one of 
California's Regional Occupation Centers before secondary school students are certified as 
having completed a program in auto mechanics. Students are tested on all aspects of the program, 
and those achieving passing scores or better are awarded certificates of completion. 

As is evident from this single example, defining program completion standards for 
several hundred different vocational education programs is a large task. Although a few states 
and some local school districts — especially those operating vocational high schools, area 
schools, or technical institutes — have developed detailed descriptions of basic job specific skills 
to be taught in each vocational program, the practice is far from universal. Furthermore, adequate 
testing of skill acquisition is not fully developed. Testing tei.ds to be strongest in programs 
preparing students for occupations requiring a license or credential. 

Because competency-based instruction is not no\K universally used and because much 
additional work remains to be done on effectively assessing competencies, adopt' g a 
competency-based vocational education system nationwide will be a major undertaking. Sikills 
must be defined in terms that are measurable and relatively free from bias, and appropriate levels 
of competency must be established for each program. None of this can be accomplished quickly 
or easily, and local schools and school districts will need substantial discretion to develop 
standards that fit their local circumstances. To accelerate and standardize the process, it will be 
tempting to establish statewide standards for each major program. At the outset, however, the 
temptation to do this should be resisted. There is too much variety among programs from one 
location to another to make this approach feasible. Nor is it clear that statewide standards would 
even be desirable. Many programs are designed to meet specific local needs, and there is no 
reason to force all vocational programs into a single mold simply to facilitate the evaluation 
process. 

To assist local districts with the development of competencies and benchmarks, states could 
develop model sets of competencies, beginning with those vocational programs that prepare 
students for occupations subject to state licensing. This is the appropriate place to start because 
there is more iikely to be consensus on what competencies are required and what level of 
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performance is appropriate. As experience is gained in these program areas, efforts may shift to 
programs where consensus is likely to be more difficult to achieve. 



TABLE 3 

BASIC AND JOB SPECIFIC SKILLS REQUIRED FOR SUCCESSFUL 
COMPLETION OF AUTO MECHANICS PROGRAM 



BASIC SKILLS 

1. Reading Skills 

a. Able to read and understand abbreviations — lbs., psi., etc. 

b. Able to read and undentand technical vocabulary — chassis, transmission, drive train, compression, etc. 

c. Know where to find technical information — specification manual, electrical wiring information, etc. 
A Able to follow simple directions requiring 1 or 2 steps (e.g., safety directions) 

e. Able to read technical information (e.g., adjusting valves, ti^iung) 

f. Able to apply information from reading to practical situations (e.g., specification/motor manuals for tune- 
up, brakes, or lube job) 

g. Able to read and interpret information on charts, graphs, or diagrams (e.g., diagrams of auto parts) 

2. Writing Skills 

a. Able to write legibly 

b. Able to write numbers 

c. Use correct spelling — general vocabulary 

d. Able to spell relevant technical terms — e.g., solenoid, carburetor, etc. 

e. Able to answer questions or fill in reports with one or two words (inventory, ordering) 

f . Able to understand and complete forms needing numbers — e.g., repair o iers, purchase orders, etc. 

g. Able to record observations 

3. Computational Skills 

a. Able to read and understand place values for whole numbers 

b. Able to read and write decinials 

c. Able to read and write simple fractions 

d. Know basic addition and subtraction (add and subtract two-digit numbers, including decimals) 

e. Able to multiply whole numbers and decimals by one- or two-digit numbers 

f. Able to divide a whole number by a one- or two-digit number 

g. Able to change a fraction to a decimal and reverse 

h. Able to add and subtract money 

i. Able to make linear measurements 
j. Able to make volume measurements 
k. Able to measure angles 

1. Able to use and understand ratios 

4. Verbal/Listening Skills 

a. Able to answer direct questions 

b. Able to follow simple and complex oral directions 

c. Able to give directions and explain procedures to otherc 

d. Able to observe and describe accurately 

JOB SPECIFIC SKILLS 

1. Skill: The student is familiar with shop safety, basic shop procedures and regulations. 

Performance standard: The student will score 70 percent or better on a test of shop safety, basic operating 
procedures, and regulations. 
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TABLE 3 (CONT.) 



2. Skill: The student is familiar with basic shop tcx)ls and equipment 

Performance standard: The participant will score 70 percent of better on a test examining the student's 
knowledge of shop tools and equipment 

3. Skill: The student can perform basic lubrication, batter, and cooiirg services. 

Performance standard: The student will achieve a score of satisfactory or better on tests of the student's 
ability to lubricate the chassis, change engine oil and filter, test and change anti-freeze, test cooling system 
pressure and check for leaks, check transmission oil, test battery, check electrolyte level, and recharge and 
replace battery. 

4. Skill: The student can perform basic tire service. 

Performance standard: The student will achieve a score of satisfactory or better on tests of the student's 
ability to change a tire, ^-^pair a flat, static tire balance, and spin tire balance. 

5. Skill: The student can perform a basic tune-up. 

Performance standard: The student will achieve a score of satisfactory or better on tests of the student's 
ability to analyze an engine with an oscilloscope, test compression, use a vacuum guage, use an ohmmeter to 
test plugs, time an engine with a timing light, use a distributor machine, use a volt/amp tester, use a coil 
tester, adjust a carburetor, and adjust valves. 

6. Skill: The student can service emission control equipment. 

Performance standard: The student will achieve a score of satisfactory or better on tests of the student's 
ability to test emission control equipment and service the smog system. 

7. Skill: The student can perform headlight adjustment and lamp replacement 

Performance standard: The student will achieve a score of satisfactory or better on a test of the student's 
ability to adjust headlighc and replace lamps. 

8. SkiU: The student can peiform basic brake service. 

Performance standard: The student will achieve a score of satisfactory or better on tests of the student's 
ability to inspect the brake system, use brake system measuring tools, reface brake drums, reface brake disc 
rotors, repair the master brake cylinder, replace a wheel cylinder, gnnd brake shoes, use a pressure bleeder, and 
use manual pullers. 

9. Skill: The student can perform basic front end service. 

Performance standard; The student will achieve a score of satisfactory or better on tests of the student's 
ability to inspect the front end, repair front suspension, align the front end, replace shock absorbers, and 
replace the MacPherso% shock struts. 
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Initially, these state models should be used for discussion purposes only, it would be 
premature to attempt to develop competencies and performance standards to be adopted 
uniformly statewide. As experience is gained with competency statements, and as the efforts of 
local school districts are assessed with regard to similarities and differences, the adoption of 
statewide standards in at least some programs may be reconsidered. 

At the local level, developing competencies and performance standards should include substantial 
assistance from employers and labor unions in defining competencies and establishing 
benchmaiics of performance that are necessary to succeed in specific occupations and industries. 
Employers must enter these discussions with the understanding that it is encumbent upon them to 
be ^Decific about their needs for basic education and job skills. Too many employers, when 
queried about the kinds of abilities they seek in employees, have been wont to reply: "Just give 
us people with basic skills. We'll do the specific job training." This response not only begs the 
question, but also, if the requirements set forth in the want ads are any indication, badly 
misrepresents what employers really want when they are seeking skilled labor. Few companies 
expect to have to teach secretaries how to type, bookkeepers how to do basic accounting, 
electronic assemblers how to solder, or prograiiimers how to use COBOL. Ytt these arc hardly 
"basic" skills that we expect all s^'ients to have in their generic toolkits. In this regard, the entire 
debate over ''basic vs. job specify" skills is misleading, for many of the skills acquired by 
students, especially as their educational careers advance, will not fit neatly into one category or 
the other, especially if these skills are taught with any eye toward their application in real life 
situations. 



CREATING FISCAL INCENTIVES TO PRODUCE PROGRAM CCfVlPLETERS 

Historically, federal funds intended to improve the quality of vocational education 
programs and to improve access have been distributed largely on the numbers of students 
enrolled in vocational education. Although other factors, such as local wealth or unemployment 
rates, have also figured in the distribution, the dominant factor has always been some type of 
enrollment measure. Under these arrangements, schools and LEas have r'^med program 
improvement and setaside funds regardless of what happens to students enrolled in vocational 
education programs. Outcomes, then, have had little or nothing to do with funding in all but a 
few states that have attempted to implement some performance standards (typically, a minimum 
job placement rate similar to the requirements of JIPA). 

The approach to performance-based evaluation developed in this chapter provides an 
alternative to enrollment-based allocation of federal funds: these funds could be allocated based 
on the number of completers produced by a school or LEA. As is currently the practice under the 
Perkins Act, the basic grant would be divided into two approximately equal amounts, one for 
promoting general program improvement and the other for improving program access for special 
populations.^ Sums available under the program improvement portion of the basic grant would 
be allocated to eligible recipients based on the number of vocational education program 
completers graduated in the preceding year. Sums available under the access improvement 
portion of the basic grant would be distributed to eligible recipients based on the number of 
students from special populations who completed vocational education programs in the previous 
year. The sums received under both portions would be additive — i.e., a special population 



*^Under current law 43 percent of the basic grant is reserved for program improvement, 53 percent for program 
access. 
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completer would earn funds under the program improvement allocation and under the access 
improvement allocation. 

Special populations would include men and women enrolled in programs nontraditional for 
their sex, racial and ethnic minoiides, disadvantaged students (including those with limited 
English proficiency), handicapped students, adults in need of retraining^ single parents and 
homemakers, criminal offenders in correctional institutions, and any other group that Congress 
might designate. As under current law, the access portion of the basic grant could be divided into 
sub-portions for each special population. 

Eligible recipients would earn funds und^r the access improvement grant for completers 
with special needs in programs with access ratios less than 1.0 for that particular special need. 
For example, an eligible recipient would earn access improvement funds for all black completers 
in programs with access ratios of less than 1 0 for black students. They would receive no 
additional access improvement funds for black completers in programs with access ratios of 1.0 
or greater for black students (they would, however, receive program improvement funds based 
on the number of total completers in all programs, regardless of access ratio). Limiting the receipt 
of access improvement funds to programs with access ratios of less than 1.0 ensures that these 
funds will not create incentives to direct students with special needs into less advanced prograir^ 
where the probability of completion is higher. 

If the allocation of federal funds were based on completers, the amount per completer 
would be quite large compared to the amounts per student available under an enrollment-based 
distribution system. For example, in 1978-79, total federal expenditures of $658 million 
amounted to only $39 per student for the 17 million students enrolled in vocational education but 
amounted to $375 per completer for the 1.75 million students completing vocational education 
programs. 

Had the federal funds in 1978-79 been divided into two equal parts, one for general 
program and one for improving access, and the funds been distributed based on completi^^ns, as 
described above, eligible recipients would have received $187.50 per completer under the 
program improvement grant and, assuming about 30 percent of the completers were students 
with special needs, about $625 per completer under the access improvement grant.^ As the 
amount per completer under this access improvement grant is quite large, ii would have provided 
a powerful incentive to increase the number of special needs completers in programs with access 
ratios less than 1.0. 

The approach to funds distribution proposed here has two major advantages over current 
policy. First, the receipt of funds depends on successful program performance, generally and 
with respect to students with special needs. Second, compared to past practices, completer-based 
allocation is simple to understand and to administer. Both features would greatly improve federal 
vocational education policy. 



^Whether 30 percent of completers in 1978-79 were, in fact, in special needs categories and whether they would 
also have completed programs in which access ratios were less than l.O for their particular special needs category 
cannot be determined. However, it is not likely that the percentage of special needs completers was greater than 30 
percent, and the assumption indicates clearly that the amounts per completer under the access improvement grant 
would be substantial. 
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CHAPTER 4 
APPENDIX 



The approach to performance-based evaluation described in this chapter focuses on 
program completion and program access (as measured by rates of completion by race, sex, and 
soeciai need) as the primary criteria for assessing program effectiveness. For purposes of 
I ^utine, on-going evaluation, this approach assumes that pro£,.am completion is synonomous 
with employability and that, therefore, programs with higher rates of complet^'on are more 
effective tha^i programs with lower rates. Periodically, this assumption needs to be tested with a 
more comprehensive analysis of program outcomes to dctermin:^ how well the required 
competencies and performance :tandards have been specified for each vocational education 
program. In ^his appendix, we sketch an analytic methodology for an in-depth assessment of 
program outct. Ties. 

The model outlined here has a.« its primary objective quantifying the relative effectiveness 
of various vocational education programs, controii'r.g for a number of contextual variables that 
influence program outcomes but are beyond the control of vocational educators and 
adniinistrators to change. In its ^nost general form, the model may be expressed as follows: 

pTJ^ram Outcomes = f(P^nqram Quality, Context) 

Table 1 lists a variety of .ncasurcs for each of these three types of variables. To bort out the 
effects of these different variables on program outcomes, we need a methodology that can 
quantify the relative contributions of the contextual and qualitative variables on program 
outcomes. 

There are a number of statistical techniques for separating the effects of causative factors. 
One of these is multiple regression. Expressed in a form that can be estimated using multiple 
regression, ^he general equation above bci:omcs: 

OUTCOMEij - ao 4 biytCONTTEXTyk + b2ijmQUALITYijn^, e^ (1) 
Where 

OUTCCMEjj = an array of outcome variables, j, for program i 
CON'^XTij^ an arrav of contextual variables, k, for program i and outcome j 
QUALITYijm = an array of quality variables, m, for program i and outcome j 
eij - an error term 

and where is a constant and h^^^ and b2ijm ^ coefficients describing the contribution of each 
of these sets of variables to the program outcome measures. 

Consider the array of variables represented by QUALITY for any particular projgram i. 
These might include such char?cteridtics as curricuL m quality, adequacy of facilities and 
equipment, class size, instructor experience and other qualifications, and so on. In effect, 
QUALITY represents a set of progranmiatic characteristics within the control of vocational 
educators that affect program outcomes. The effects of these characteristics are what the model 
seeks to isolate from the noncontrollabl^ contextual variables. 
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While some of these progr?minatic characteristics, 5>uch as class size and instructor experience, 
can be measured, some of the most important ones, such as curriculum quaJity, cannot. Using 



TABLE 4 

Measures of Program Outcomes, Contextual, 
and Qualitative Variables 



PROGRAM OUTCOMES 

Placement in a job related to training 
Placement in another job 
Length of time unemployed 
Earnings 

Acquisition of specific occupational skills 
Acquisition of basic skins 
Further education or training 

Reduced stereotyping with respect to race, sex, or special need 

CONTEXTUAL VARIABLES 

Characteristics of Students 

Scholastic Aptitude 
Socio-economic background 
Race 
Sex 

Handicapping condition 
Engli&;h speaking ability 

Cnaracteristics of the Delivery System 

Institutional setting (comprehensive high school, area srhool. community college, 
etc.) 

Geographic location (urban, suburbari. rural) 

Economic Conditions 

Local, regional, and national unemployment rates 
Relative growth rates in occupations related to training 
Relative wage levels in occupations related to tcaining 

PROGRAM QUAUJY 

Experience of instructor 

Condition ot supplies and equipment 

Expenditures per student 

Student/teacher ratio 

Level of performance standards 



60 

7(i 



partial measures of program quality could produce seriously inaccurate results and provide 
incentives to institutions to do well on aspects that arc measured, which may not be the most 
important Furthermore, program quality depends not only on these individual factors, but on 
how they are put together. For example, large class sizes are not necessarily inferior to small 
ones. What is more relevant is that the number of work stations for students match the number 
of students so that students do not have to wait for access to equipment. 

Were the model concerned with fully explaining the variation in effectiveness among 
programs, it would need to have quantitative measures for these programmatic characteristic*^ 
The objective, however, is only to isolate the effects of those variables beyond the immediate 
control of program deliverers from those over which they do have control and which, in sum, 
they may be held accountable for finding an effective mix. What we arc concerned with is the 
combined effect of the variables that can be manipulated by program deliverers. How they 
manipulate individual variables is of no concern in this model (although this 'S a perfectly 
legitimate and important conccm of other types of research efforts). It is concerned only with the 
results that have been achieved. Consequently, estimating these programmatic effects requires 
manipulating equation (1) so that QUALTTYij becomes tlie dependent variable 

b2ijQUALITYij = OUTCOMEij - (ao + bujtCONTEXTijk +Cij) (2) 

QUALITY, then is what is left when the effects of the contextual variables are subtracted 
from the OUTCOME measures. The outcomes can be estimated in terms of only contextual 
variables as follows: 

Oli 1 v^OMEij = ag + bujtCONTEXTyk + ey (3) 

Using ihc regression coefficients resulting from equation (3), the expected outcome, 
OUTCOMEij* given the contextual effects, can be computed for each program i for outcome j in 
each district: 

OUTCOMEij* = ao + b2ijLC0NTEXTijk (4) 
Then, 

QUAlJTfij = OUTCOMEij -OUTCOMEij* (5) 

QUALITYij, therefore, computed for each district, provides an estimate for each program i, 
the qi»ality of that program with respec' to the outcome variable j. The absolute value of 
QUALITY would not have any real-world meaning, but the value is an indicator of the relative 
effectiveness of a program (with respect to a particular outcome) in one district compared with 
the same program in another district. A large value for QUALITY would mean that there was a 
big difference between the actual outcome and the outcome than would be expected on the basis 
of the contextual characteristics alone. This might occur, for example, in a program that had a 
large number of disadvantaged students who would normally not be expected to have highly 
successful outcomes, who in this case did very well because of an exceptional instructor who 
carefully tailored the material to the students. 

A model constructed along these lines would permit several types of analysis to aid state 
and local planning. Firsr, for any particular type of vocational education program (electronic 
technology, for example) it would permit statewide ranking of program offering and 
identification of those districts which appeared to be offering the most effective programs (say 
Uiosc in the top twentieth percentile) or the least effective programs say those below tho twentieth 
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percentile). Second, by computing statewide or regional averages, it would permit a state to rank 
programs by type, comparing the relative effectiveness of, say, welding versus carpentry or 
masonry. Third, the model would permit districts to see how their program offerings performed 
relative to one another and relative to others in the state or nearby districts. Such information 
would also aid consumers to vocational education, who have options among offerings of local 
school districts and those provided by conmiunity-based organizations or proprietary schools. 
Similarly, such information could assist PICs in tiie selection of program deliverers for JTPA. 
Fourth, the model would permit a number of additional inquiries, including analysis of cost- 
effectiveness, differences among different kinds of delivery systems, the influence of certain 
student and district attributes on program effectiveness, and so forth. Tliese are topics that have 
been much discussed in vocational education, but that to date have eluded systematic analysis. 
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CHAPTER FIVE 



IMPROVING VOCATIONAL EDUCATION DATA 



If the pcrfomancc-bascd approach to evaluation developed in the previous chapter is to 
provide vocational educators and policy makers with useful finJinjs, it must use data that are 
accurately and consistently reported. Unfortunately, accurate and consistent data for vocational 
education have eluded federal policy for over two decades, despite several serious and costly 
attempts to improve the information. By fai' the largest effort was the establishn^ent of the 
Vocational Education Data System (VEDS) by the 1976 Amendments to the Vocational Education 
Act. VEDS was intended to generate uniform data on vocational education students, p/ograms, 
program outcomes, staff, facilities, and expenditures. Congressman Carl Perkins, Chairman of 
the House Committee on Education and Labor, sunmiarizcd the general purpose of VEDS in his 
opening remarks on VEDS durii.g hearings on December 10, 1981: "In mandating this system. 
Congress was respondii.^, to a lack of adequate data to judge program effectiveness and to make 
important decisions about future directions." 

VEDS began collecting data in 1978-79 and continued collecting data for four years. No 
one knows how much VEDS cost — some estimates have been as high as $200 million when the 
resources expended by state and local personnel were counted — but by 1983, the realization 
was growing that the latest attempt to improve vocational education data had failed. In 1983, the 
National Center for Education Statistics, the federal agency responsible for the design and 
administration of VEDS, performed a number of validity tests on the four years of accumulated 
data. These checks confirmed what many critics of VEDS had argued from the outset: three 
major problems plagued the system: 

1. Lack of comparability among states. Data were not comparable from state to 
state and, therefore, yielded misleading national totals v/hen aggregated. 

2. Year-to-year variability. The data exhibited excessive variation over time, which 
was difficult to explain. Consequently, the VEDS data could not be used to describe 
trends over time accurately. 

3. Within state discrepancies. When VEDS data from some states were compared 
to state data from other sources, many large discrepancies were observed; these could 
not be adequately explained 

Following these findings by NCES, the Office of Management and Budget (0MB), in 
December 1983, withdrew approval for collecting VEDS data for 1983-84 and 1984-85, because 
''VEDS has substantial and continuing problems collecting data which are accurate and 
meaningful." 0MB directed that the "collection should not be undertaken at this junction and 
should remain suspended until the Department [of Education] has fully implemented the plan for 
improving VEDS." 

In February- 1984, the NCES Administrator, with the concurrence of Office of Vocational 
and Adult Education, made the following statement about VEDS in a memo to the Secretary of 
Education: 

The current [VEDS] system has serious technical probleirs. NCES believes that 
the technical problems do not lend themselve:i to correction without a major 
system redesign. The data are unreliable and subject to serious misinterpretation. 

VEDS was dead. 
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The failure of VEDS did not dissuade '^ongress from trying once again. Section 421 (a)(1) 
of the Carl Perkins Act directed NCES to develop, in consultation with the Congress: 

a national vocational education data reporting and accounting system 
using uniform definitions. The system required by this section shall 
include information on vocational education — - 

(A) students (including information concerning race, sex, and 
handicapping condition) 

(B) programs, 

(C) program completers and leavers, 

(D) placement and follow-up, 

(E) staff, 

(F) facilities, and 

(G) expenditures in relation to tlie principal purposes of this Act 

Additionally, Section 423 of the new law directed the Secretary of Education to ensure that 
adequate information is collected on the access of secondary handicapped students to vocational 
education programs. It stipulated that data on handicapped enroMment were to be reported 
bienniaUy by "instructional setting" and "handicapping c :>ndition," in "4-df git detail as defined in 
A Classification of Instructional Programs (CEP)" published by the National Center for Education 
Statistics 

The new dati reporting requirements differ from VEDS in several important respects. 
First, to improve the quality of data and to reduce the burden of collecting it, the system is to rely 
heavily on sampling rather than surveying the universe of providers of vocational education 
programs. Second, the national collection is no longer confined to using data collected by the 
states. Third, instead of annually collecting all vocational education data, the system is to be 
updated no more frequently than every two years. 'egislation gives wide Discretion to the 
Secretary of Education, who in consultation with Con^ ess is to determine tfie number and types 
of vocational education institutions to be sampled, the appropriate methodologies, sample sizes, 
appropriate analyses, and the frequency of reieva-^t studies. 

Despite these changes, however, it is likely that NCES will still encounter significant 
problems in responding to the Congressional mandate. There remain basic misunderstandings 
about the difficulties associated with collecting vocational education data, and until these can be 
resolved. Congress will continue to ask for information that, at reasonable cost, simply cannot be 
collected accurately. It is important, therefore, for policy makers to understand what data can be 
obtained and how they can best be collected. The rest of thi*^ chapter outlines our conclusions 
about the feasibilitj' of collecting various types of vocational education data and outlines a new 
approach to collecting vocational education data that tries to avoid the major problems 
encountered with VEDS. 



LESSONS FROM VEDS 

• The data required for accountability and policy cannot be satisfied with a single, annual data 
collection. 

VEDS failed to distinguish between data required for accountability and data required for 
policy making. It adopted a single strategy for doing both and as a result could do neither. In the 
view of one observer: 
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The reason VEDS failed to provide good information is that it was not designed 
with that end in mind. Instead, VEDS was designed as an exercise in 
accountabilty and compliance to produce official numbers corresponding to a 
wide range of Federal purposes and concerns.... the system never held any 
promise of yielding information.^ 

One of the major shortcomings of VEDS, and indeed of the authorizing legislation as well, 
was the failure to distinguish carefully the kind of information needed for purposes of 
accountability from the kind of information needed for purposes of policy making. The data 
necessary to ensure that recipients of VEA funds are conforming to the official requirements of 
the law need to be maintained annually by all recipients For compliance purposes, sampling 
does not suffice; each recipient needs to maintain sufficient, auditable information. Ideally, 
because these data requirements must be imposed universally, they should be as simple as 
possible and limited to the bare essentials for program accountability. Moreover, it is rare that all 
data needed for accountability need to be report, as long as the information is maintained on 
site for possible audit or review by the interested public. 

Satisfying the requirements for addressing major policy issues involves more detailed and 
more complex data. However, policy-related data may usually be collected less frequently and 
from samples of respondents, greatly reducing the burden that would be imposed if these needs 
were to be met by the methods used to collect data for purposes of accountability. 

A better distinction between accountability requirements and policy requirements for data is 
needed to resolve ^me of the major problems that have plagued the collection of national data for 
vocational education. For example* what constitutes vocational education and who is a vocational 
education student are two questicms that have generated considerable controversy and confusion. 
VEDS took the position that only students enrolled in programs approved under the State Plan, 
and therefore eligible for VEA funds, were to be counted. For purposes of accountability, this 
position is eminently sensible. Congress wants to know how many students directly benefit from 
VEA funds. Moreover, requiring recipients of federal funds to determine and report the number 
of students served is basic to sound orogram management 

For policy purposes, however, ti*:., decision leaves much to be desired. As long as states 
enjoy considerable discretion in determining what programs are eligible for VEA funds, the 
"State Plan universe" will fall far short of describing the total vocational education enterprise.^ 
To assess the extent of coverage achieved by VEA funds, to evaluate the adequacy of the labor 
supplied by various training institutions, and to assess the relationship between vocational 
education and other aspects of the secondary and postsecondary system, there is a need for 
information that describes with reasonable accuracy the nature of the overall enterprise. 



^Robert E. Barnes, "Why VEDS Failed," Office of Planning, Budget, and Evaluation, U.S. Department of 
Education-, February 1984, mimeo, p.2. 

2Under the new legislation, the "State Plan universe" is even more likely to pose problems for adequately 
describing the vocational education enterprise. Many states plan to distribute federal vocational education funds on 
the basis of proposals from individual LEAs rather than on the basts of formulas. Consequently, many LEAs with 
vocational education programs will not receive federal funds. Moreover, the identity of LEAs receiving funds will 
change from year to year. Therefore, dau collected only from LEAs receiving federal funds will serve 
accountability t.iterests only; for purposes of addressing basic policy issues, such data will be useless. 



VEDS tried to satisfy data requirements for accountability and policy with a single annual 
census, conducted by the states, of all recipients of VEA fvnds. The fine detail required for 
policy purposes proved too much for the states and LEAs to report accurately and efficiently. 

• Any attempt to collect detailed national data on vocational education enrollment using an annual 
census approach will continue to si^erfrom problems of inaccuracy y inconsistency, and lack of 
comparability. 

Traditionally, efforts to determine who is served by vocational education have concentrated 
on attempting to count the number of students eru'olled in vocational education during the course 
of the school year. These efforts have sought an unduplicated count of students and have also 
sought to identify enrollment by specific types of vocational education programs. Such a 
seemingly simple task is in fact quite complex and may even be impossible to accomplish 
consistently and accurately, if one attempts to do so through an annual cross-sectional survey. 
There are several reasons for this difficulty. 

First, at any single point in time, it is not easy to determine who is a vocational education 
student and who is not. At both the secondary and postsecondary level, students enroll in 
courses, not programs, and for most school systems, courses constitute the basic accounting unit 
for purposes of attendance, scheduling, grading, and reporting. However, many courses, 
particularly at the introductory level, ait taken both by students who intend to pursue a vocational 
program and by those who do not Therefore, enrollment in a particular course is often not 
sufficient to identify a student as vocational. Moreover, many vocational education students, 
especially at the post-secondary level, take more than one vocational education course during the 
academic year. Because cocoes, not programs, are the basis for maintaining attendance, 
obtaining unduplicated counts of vocational education enrollment is problematic. Consequently, 
most schools have been hard pressed to count vocational students accurately. 

Requiring enrollment by specific program compounds this problem. A program is a 
sequence of courses, and although the Classification of hstructional Programs (CIP) provides a 
uniform definition for some four hundied or more vocational education programs, there is no 
uniform definition of the sequence of courses that constitute a particular program. Moreover, one 
course may serve several different programs, making it impossible to assign a student accurately 
to a particular program until the full sequence has been completed. Consequently, a count takfcn 
at a particular point in time, forces arbitrary assignment of students to particular programs, 
leading inevitably to inconsistencies in reporting at different points in time. 

The problem can be easily illustrated. Consider students taking "Typing 1." In most high 
schools. Typing 1 is part of at least three different vocational education programs — Business 
Data Processing, Secretarial Training, and Typing and General Office (each of these has a 
separate four-digit CP code). Moreover, Typing 1 is also taken by large numbers of "academic" 
students who are not pursuing a vocational education program at all. As there is rarely any data 
available on students' long range intentions, accurately assigning students enro^^ed in Typing 1 to 
a specific vocational education program is impossible.^ 



^This problem is not solved by a collection strategy that asks students their intentions. Not only is such an 
approach very burdensome, but also it is not likely to yield accurate information. Exptiience with past surveys 
indicates large discrepancies between what students say they intend to do in their educational careers and what they 
actually do. Additionally, when asked what type of program they are enrolled in, there also large discrepancies 
between the percentage of students who say they are vocational and the percentage of students who can be classified 
vocational according to coune data from ttieir transcripts. 
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Although the problem of accurate program assignment is *iiost severe in the business 
programs, difficulties also exist in other vocational program areas. Agricultural Science and 
Agricultural Mechanics are required courses for a variety of different agricultural programs. 
Health is the beginning course in most health programs. Carpentry is the first course in several 
different woodworking programs, and Electricity is required for electronics programs and 
electrician programs. In short, accurately assigning students to specific vocational programs 
requires historical data on the sequences of courses different students have taken. Moreover, as a 
program often cannot be uniquely identified until enrollment in the terminal course of the 
sequence, accurate assignment to a four* or six-digit program code is most accurately 
accomplished at the time of program completion. 

Further compounding the problems of accurately reporting enrollment data was the 
requirement that the information be reported by race, by sex, and by special need (e.g., 
handicap, disadvantaged, limited English, etc.). The only accurate source of information on a 
student's race or ethnicity and special needs is the Individual student record and transcript. 
However, if records and transcripts arc not automated, sorting students by program by race by 
sex and by special need c2.*«not be done without unreasonable effort in any but the smallest 
schools. Moreover, even it transcripts are maintained on computer, the need for arbitrary 
decisions is not avoided. 

The enrollment data sovght by VEDS were affected by all of these difficulties and 
consequently exhibited numerous problems: 

• Most states could not produce accurate unduplicated counts of vocational education 
students enrolled by program; 

• What constituted a particular program differed £ mong LEAs, among states, and over time; 
consequently, enrollments by program could not be easily compared; 

• Reporting by race by sex and by special need was highly inaccurate. 



In short, it is not likely that accurate, comparable, detailed data on enrollment in vocational 
education can be collected nationwide from all providers. Nor can the problem be solved by 
limiting the collection to recipients of VEA funds. In part, this problem results from the lack of 
uniform standards on what courses constitute a particular vocational education program. 
However, even if uniform standards could be established (and it is by no means clear that such 
uniformity is either feasible or desirable — see discussion of uniform definitions below), the 
decision about when a particular student becomes a "vocational" student and the determination of 
the program in which tht student is enrolled would remain arbitrary. Consequently, any attempt 
to collect detailed national data on vocational education enrollment using an annual census 
approach will continue to suffer from problems of inaccuracy, inconsistency, and lack of 
comparability. 

Even if the definitional ^^d identification problems could be solved, the collection of 
enrollment data appears to serve ^o major national interest Regardless of the problems described 
above, a simple but fundamental question needs to be addressed: what national purpose is served 
by continuing to collect detailed information on vocational education enrollment on an annual 
basis? As a measure of the number of individuals with competencies to perform in specific 
occupations, enrollment in a particular program is a poor measure* Even if the definitional and 
identification problems could be solved, enrollment figures convey little information about the 
occupational competencies acquired. In this regard, program completion is a much better 
' idicator. As a measure of progranmiatic activity^ unduplicated counts are also inappropriate. 
Courses and programs differ widely in content and duration; moreover, there are important 
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qualitative differences among different delivery systems. Although the extent of program activity 
is an appropriate concern of national policy, such activity is better measured by contact hoUiS or 
average daily attendance than by unduplicated enrollment. Consequently, despite the 
longstanding emphasis on collecting unduplicated enrollments by type of vocational education 
program, the collection appears to serve no major national interest, for purposes of policy or 
accountability, even if it could be done accurately, which it cannot. 



• The classification scheme for reporting vocational education data has tob? re-examined. 

Assuming the two major policy uses of detailed program data are manpower planning and 
improving program access, two-digit reporting, which classified vocational education programs 
into eight different categories, has never been very satisfactory. The distinctions are ' ^'inc 
enough to be useful. At the other extreme, six-digit reporting would impose a large bui . . on 
institutions, especially those without fully automated student record systems. With over four 
hundred six-digit codes, reporting students by race/ethnicity (six categories) and by sex, an 
institution would have to distribute students over more than 3,600 individual data cells. The 
problem is further compounded if information is sought on handicapped students by type of 
handicap (1 1 categories) and instructional setting (3 categories)."^ 

Four-digit reporting would seem to offer a compromise, but with a 120 codes, the number 
of cells can still multiply rapidly.^ Moreover, for some occupations, the old four-digit O.E. code 
fails to provide sufficient detail. For example, carpentry, electricity, masonry, and plumbing and 
pipefitting arc aU included under a single four-digit O.E. coue. The new CIP code uses about 120 
four-digit codes to describe vocational education, and these can be further broken & vvn into 
slightly more than four hundred six-digit codes. However, CIP is far from universally 
implemented. VEDS sought to address this problem by collecting data on a mixture four-digit 
and six-digit O.E. codes. It sought information on programs in which more than 6,00 J students 
were enrolled nationally, programs within each state that accounted for at least one pc. r^ni of 
occupationaUy specific enrollment, programs identified by BOAE (now OVAE) as having speci«:1 
requirements for reporting by program, and a single category to capture "all other.** This resulted 
in 1 16 specific four- and six-digit codes, a total roughly equal to the number of new four-digit 
CIP codes. At the secondary level, however, 116 programs far exceeds the typical number of 
specific offerings, and the burden of program reporting remains significant. 



^Those accustomed to thinking of data burden in terms of the ^^^mber of variables that must be maintained will be 
puzzled bv tlie notion that the number of data cells poses a particular problem. After all, in this example, data for 
an individual student must be maintained on only five variables — program, race, sex, handicapping condition, and 
instructional setting. Most schools, however, do not have automated student record systems. Consequently, the 
typical approach to collecting dau has relied on asking respondents to complete forms that must contain all 
possible combinations of the five variables. This leads to matrices with an enormous number of cells. The forms 
are difficult to complete because most of these cells will be empty for any one respondent 

^For example, scrictly adhering to the new requirements of Section 423 for collecting data on handicapped students 
in vocational education produces 3,960 cells (120 programs x 11 handicappmg conditions x 3 types of 
instructional setting - 3,960). 
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• Limiting data collection to the State Plan universe does not provide the irformation needed for 
policy purposes. 

VEDS distinguished between secondary and postsecondary institutions, and at the 
postsecondary level, it further distinguished between regionally accredited, state approved, and 
other postsecondary institutions. Because VEDS limited collection lo the State Plan universe, 
however, it did not seek data from proprietaiy schools or other institutions providing vocational 
education not included in the state plan. Furthermore, at the secondary level, it did not 
distinguish among programs offered in comprehensive high schools, vocational high schools, or 
area vocational schools. 

Limiting the collection to recipients of federal funds is useful for accountability purposes 
only, and has no utility in and of itself for manpower planning or other policy making. 

• Procedures for following program completers and leavers m u be completely redesigned to 
improve response rates and to increase the length of time students are followed. 

In previoiis chapters we have discussed in detail the problems surrounding the definition of 
vocational education program outcomes, and we will not repeat ourselves here. Suffice it to say 
that VEDS was unable to obtain good comprehensive follow-up data, encountering many 
problems. Among them were the following: 

• Identifying program leavers was not straightforward. Transfers and temporary 
withdrawals (especially at the postsecondary level) made it impossible to count leav<;rs 
accurately at any point in time. As a result, the data on leavers were highly suspect. 

Locating students, especially postsecondary students, after leaving or completing a 
program was difficult and expensive; as a result, follow-up data contained unacceptably 
large numbers of students whose status was unknown. 

• Obtaining sufficiently high response rates from students and employers was expensive 
and beyond the resources of most states to accomplish on a large scale; in most states, 
response rates of 20 to 30 percent for students and even lower rates for employers were 
common, making the data unusable. 

• VEDS follow-up was limited to a period of about six months following leaving or 
completing a program; this is too short a time to assess adequately the effects of 
vocational education, but follow-up for longer periods of tir^e was too expensive to 
pursue on a large scale. 

In short, effective larg^ scale follow-up of leavers and completers is not possible at reasonable 
cost. Nor have attempts to limit follow-up to completers or to permit sampling 20 to 25 percent 
of LEAs per year been effective. 



• Tracing federal dollars programmaticaliy is virtually impossible. 

Congress has asked a number of questions about the finances of vocational education. 
These include: 

• How much state and local money is expended for vocational education? 

• How much is spent on vocational education for students with special needs? 
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• What are the "excess cc s" required to improve the access of students with special needs 
and for what are these cos^^> incurred? 

• What are the relative costs of different programs and different delivery systems? 

• How is federal money allocated among eligible recipients? 

• What do federal VEA funds buy? 

Some of these questions are easier to answer than others. Although determining what is 
spent for vocational education may seem straightforward, it is in fact quite difficult. At the state 
level, most SEAs can easily keep track of the allocation of federal VEA funds to recipients, but 
with the exception of those few states that provide separate state funding for vocational 
education, most states cannot determine how much state money is spent for vocational education. 
State funds for vocational education are allocated as part of general assistance to LEAs, and 
consequently, there is no separate accounting. 

Similarly, at the local level LEAs maintain accounts that distinguish revenue by source 
(federal, state, and local) and expenditures by object (salaries, benefits, supplies, equipment, 
etc.). Relatively few LEAs maintain accounts of expenditures by program. Therefore, they do 
not know precisely how much state arid local money is spent for vocational education. Estimates 
can be and are made, but these are subject to considerable error and variability over time. 

If it is difficult to determine how much is spent for vocational education as a whole, then it 
is even more difficult to determine how much is spent by type of vocational education program. 
VEDS sought expenditures by two-digit program code, but the information submitted was highly 
suspect, often estimated simply on the basis of the distribution of enrollment among the different 
programs. Even if this prorating were an acceptable procedure for estimating expenditures — and 
this is not likely — the underlying enrollment data were inaccurate and therefore could not 
possibly lead to accurate estimates of expenditure by program. 

Even if accurate estimates of expenditures could be obtained by two-digit code, it is not 
clear what could be done with the information. The two-digit distinctions are too general to 
permit any useful analysis of differences in costs and expenditures among programs. Analysis at 
the four- or six-digit level, which could be useful for resource allocation policy, is simply 
beyond the capabilities of most LEAs and SEAs on a regular basis. 

A related problem is the calculation of "excess costs" associated with providing services to 
students with special needs. Funds allocated ander the setasides for handicapped and 
disadvantaged students, in both P.L. 94-482 and P.L. 98-524, are to be used to pay for not more 
^han one-half of the funds expended abo/e average expenditures per student for students with 
special needs. In practice, most LEAs have found it impossible to comply with this rcquiicment. 
Not only do they not know precisely what average expenditures per student are for vocational 
education, but also their accounting systems are not designed to keep track of additional 
expenditures on certain types of students or programs. Moreover, for most LEAs, designing an 
accounting system that would do so is a costly and complicated task. 

Finally, it should be noted that while it is easy to trace the allocation of federal VEA funds 
to an LEA, it is difficult to determine how federal funds alone are spent, because it is difficult to 
avoid commingling with state and local revenues. Even separate accounting for federal revenues 
does not really solve this problem because of the "substitution effect.'* For example, suppose 
federal VEA funds are used to buy computing equipment. One has no way of knowing whether, 
in the absence of federal funds, the local recipient would have bought the computing equipment 
vvith state and local dollars. If the recipient would have used state and local dollars and simply 
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substituted federal dollars for a purchase that would have been made anyway, then the federal 
dollars did not enable the purchase of the computing equipment but rather enabled the purchase 
of whatever was bought with the state and local dollars that otherwise would have been spent on 
computing equipment Sorting out these substitution effects and establishing a direct causal link 
between the receipt of federal funds and expenditure for a particular purpose is very difficult. 
Therefore, attempting to isolate the effects of federal expenditures has not yielded useful 
information. 



• It is not feasible to collect staffing data by program on an annual basis. 

VEDS initially sought to collect staffing assignments to vocational education programs by 
race by sex, but subsequently suspended the staff report. Many of the same problems that arise 
with identifying a vocational education student arise with identifying vocational education staff. 
In most LEAs, both secondary and postsecondary, personnel are assigned to departments and 
teach courses, not vocational education programs. Many staff teach courses that arc taken by 
both vocational and non-vocational students. Consequently, allocating their time is problematic 
and subject to confusion and error. 

Despite these problems, the collection of staff data has many attractive features if it could be 
done accurately. First, as a measure of activity and resources in vocational education, staffing 
FTE is a much better indicator than student enrollment or prog ram completers. With good 
information on expenditures by program unavailable, staffing FTE probably provides the second 
best indicator of expenditures by program. Moreover, there are far fewer staff than students, 
which presumably should make the data easier to collect Second, data on staffing characteristics 
could provide some good indicators of differences in program quality — between vocational 
education and other education programs, as well as among different vocational education 
programs and different delivery systems. To be useful, however, such data would need to be 
rather detailed, making it infeasible to collect on an annual basis 



• The search for uniform definitions may be a hopeless quest. 

To some extent, the many problems confouncUng the collection of accurate data on 
vocational education come down to this: the 50 states and the 10,000 or so LEAs and 
postsecondarj' institutions (PSIs) with vocational education programs all do things differently. 
Courses have different titles and are taught for different amounts of time. Course content varies, 
and the sequence of courses that make up a particular program differs among LEAs, PSIs, and 
states. Requirements for program completion also vary, as do procedures for program approval 
and teacher certification. States do not all ase the same procedures for defining and identifying 
disadvantaged students and others witli special needs. Althoug^ The Education for All 
Handicapped Children Act (P.L, 94-142) has established definitions of handicapped children that 
are routinely used by most secondary schoo-s, postsecondary institutions use Oifferent 
definitions, and many do not systematicaUy identify handicapped students at all 

Amidst such chaos, it is tempting to believe tha^ many of the data problems could be solved 
by universally adop>ng uniform definitions, and, indeed, both P.L. 94-482 and P.L. 98-524 
have required the u c of jniform definitions in national vocational education data systems. It is 
important to undf. stand why — with the possible exception of defining students with special 
needs — uniform, specific, '^rational definitions are not likely to improve our understanding of 
the vocational education enicrprise. 

In large measure, v national education programs are effective c y to the extent that they 
are designed to meet loral and regional labor market needs. As these i *eds vary, depending on 
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local technology and the degree of local labor market specialization, programs often will be 
effecti/c only if they are not uniform in content or in the specification of competencies required 
to fill an entry level position. As we noted in Chapter Four, an automotive program designed to 
train students to work in rural arras may stress a broader range of general sluUs, using less costly 
and technically sophisticated equipment, while an automotive program in a major city may stress 
greater specialization on more complex diagnostic and repair equipment. Imposing uniform 
standards for program content or program completion could easily produce the result that 
students in neiUier area would be effectively trained to enter the kinds of jobs most immediately 
available to them. 

Similar problems can arise in efforts to define uniformly students with special needs. For 
example, uniform income criteria for defining economically disadvantaged students will lead to 
understating the numbers of disadvantaged students in areas where the cost of living is high and 
overstating the numbers in areas where the cost of living is low. Attempting to design uniform 
procedures for adjusting definitions for cost of living differences is likely to encounter many 
difficulties. 

Leaving states ard LEAs to their own devices, however, raises other kinds of problems. 
Without any direction, states may be tempted to define special needs criteria as generally as 
possible to permit spreading restricted funds over as many students and LEAs as possible. 



A NEW STRATEGY FOR COLLECTING VOCATIONAL EDUCATION DATA 

Keeping in mind the irrfoimation needs for both accountability and policy and the problems 
^rxountered in using VEDS to try to meet them, we propose a ncA^ approach to collecting 
vocational education data. In place of VEDS, we propose two types of data collection: 

• periodic surveys of small national samples of students and institutions; 

• an annual or biennial universal census collecting a limited amount of data. 



Periodic Sample Surveys 

Much of the data required for policy purposes can be obtained accurately only by collecting 
historical data on students' course taking patterns and on their status following program 
completion or leaving. Such longitudinal studies can avoid many of the problems V^DS 
encountered in attempting to collect vocational education data. Because they develop a history of 
cor-^es taken by each student in the sample, students may be accurately and consistently 
assigned to the proper vocational education program. Accurate unduplicated counts of students 
tnrolled in vocational education during the year may be obtained, and patterns of program 
leaving and completion may be carefully examined. The studies generate excellent follow-up data 
on employment arid other program outcomes, and these data are collected over a period of at leasi 
four years, allowing reliable comparisons between stuaents enrolled in vocational edu ation 
programs and those who are not. The data also permit analysts to control statisticall> ^or a 
number of variables that may affect patterns of program enrollmen and program outcomes — 
e.g., race, sex, special need, parental and family characteristics, school characteristics, etc. 

In addition to producing much more ^^.curate and reliable data than the approach used by 
VEDS, lon£;itudinal studies are much less burdensome. The collection is limited to a sample of 
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schools and students, and the responsibility for collecting the data is assumed by a federal 
contractor rather than the schools and state agencies. 



The logical vehicle for collecting this kind of information is the longitudinal studies already 
being conducted by the Nalional Center for Education Statistics. NCES has previously conducted 
two longitudinal studies, the National longitudinal Study (NLS-72) and High School and 
Beyond (HS&B), which have yielded extensive information on vocationaJ education. For NLS- 
72, base-year data were collected on a cohort of high school seniors in 1972. 5!it^ce then, that 
cohort has been followed up four times. Postsecwnd^ transcripts ivere collectea ^n 1984, and a 
fift^ follow-up is schedul<*d for 1986. For HS&B, base-year data on a cohort of sophomores and 
a cohort of seniors were collected in 1980. Two follow-up data collections have already 
occurred, and two more are scheduled, nne in 1986 and another in 1990. Additionally, 
postsecondary transcrip"j were collected for the senior cohort in 1984 and will be collected for 
the sophomore cohort in 1986. Both NLS-72 and HS&B provide a rich source of existing, 
usable data for analyzing such topics as the number of F*udent5 takif^ vc<;ational education 
courses, patterns of course taking throughout students' educational r-'reeii, psttejns of program 
completion and leaving, and a var y of program outco^nes (empl. ^ mcnt nisiory, patterns of 
further education and training, cz^ ingi, etc.). 

A third longitudinal study, the National Eiucation Longitudinal Study (NELS), is slated to 
begin with a secondary cohct and a postseconduy cohort in spring 1988. At least two follow-up 
collections are planned, one ii* 1990 and another in 1992. NELS can easily be modified to collect 
more detailed data on vocational education, including students, types of delive^' ./stems (e.g., 
comprelicrsive high schools, vocational high schools, and area vocational schools), staff, and 
facilities. 

Despite thei/ power to generate ga)d data on vocational education, these longitudinal 
studies h^ve a number of limitations. First, they are able to generate geographically specific data 
for only uic nine U.S. Census Divisions and about eight states. States are ^tncouraged to augment 
the sample, at tJieir own expense, to generau ^tatc-specific data, ana scv era: have chosen to so in 
the past twc collections, Hovvcver, augmentation is most costly in the smallest sta^^s, which 
usually are the least able to affoid it Second, the sample of students i<^ not large enough to obtain 
a great deal of det^l by type of vocational education program. Typically, 15 to 25 of the largest 
vocational education programs can be differentiated, but ti\is is far short of the 120 four-digit and 
400 plus six-digit CIP classifications for vocational education. Third, the sample size is not large 
enough to permit analysis of program partricipation by special educatio; stii Jv-nts, nor is it 
possible to oversample spec: d education students at reasonable cost, P "^rtli, ii*jre is a six to 
eight year lapse between Uie selection of new cohorts, so that the data arv , always as timely as 
one would like. 

This lack of geographic specificity, progi'tm detail, special education data, and ongoing 
timeliness limits the usefulness of the longitudinal studies for purposes of manpower planning, 
especially at the >tjie and local level, it aiso limits their usefulness for detailed and timely 
excminat^'^ns of program access Ij race, sex. and special need. Consequently, t^ie data collected 
through the longitudinal studies need to be supplemented with data on progrs^r i completers and, 
if necessary, program enrollment collected more frequently as part of an ainual or biennial 
universal census. 



Annual or Biennial Universal Census 

The census should concentrate on collecting data on program completers. All providers of 
vocational education should be surveyed and asked to report the number of program completers 
by race, sex, and handicapping condition by type of vocational education program. 
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In our model for perfr mance-based evaluation we defined program completers as those 
students "satisfying the requirements for a degree, certificate, diploma, or other formal award 
and complelinfi; a vocational education program that certifies that the student has acquired 'he 
general and job-specific skills necessary to perform effectively an entry level job in an occupation 
related to the student's training." We would leave states and LEAs free to use their own 
discretion as to how to operationalize this general definition, based on differences in local labor 
market conditions and approach to curriculum. These procedures, however, should be subject to 
review. 

At the secondary level, probably 40 to 60 different program categories would piov^ide 
sufficient programmatic detail, although this issue would benefit from more thorough study. To 
some extent, the problem of reporting with a high degree of program specificity is a problem of 
instrument design and collection strategy. While it is true that, at the level of sixnligit detail, there 
may be several hundred different vocational education programs within a state, a specific 
institution typically offers no more than a dozen six-digit programs, and most of the LEAs offer 
no more than thirty. Consequently, if the program data could be collected and tra^^^mitted by 
institution or LEA, it would be possible to use survey instruments that permit respondents to list 
only progrrims relevant to them. After the initial data collection, subsequent surveys could 
employ a "shuttle form" with the respondent's programs from the preceding survey preprinted. 
The respondent would need only to note changes. For such a strategy to work, however, the 
underlying classificatiDn system (CIP) .^ust be sound and employed universally. Otherwise 
various types of translation procedure?; ("crosswalks") must be used to standardize program 
offerings, introducing increased potential for error. 

For manpower planning, and for other policy purposes, it is clear that completion data are 
needed from all insututions providing vocational education, regardless of whether or not they 
receive VEA fvnds. Moreover, given the rather large qualitative differences that exist among 
prograni. d':^. ling upon whether they are offered in comprehensive high schools, vocational 
high schools, area vocational schools, vocational-technical institutes, private proprietary schools, 
and other postsecondary institutions, it would be useful to have completion data reported 
separately for these different institutional types. Furthermore, within the public sector offerings, 
it would be useful to have program completion data reported by 'ice, sex, and special need for 
these different institutional types. As proprietary schools not receiving federal funds are beyond 
tlie reach of federal policy on access issues, it may prove impossible to obtain data on race, sex, 
and special need for these institutions. 

Making these kinds of distinctions among different types of volitional educatio providers 
would be most easily accomplished if the data were collected and transmitted to the f .^al level 
by institution or LEA. Disaggregated data would '^Iso facilitate error ''hecking. However, 
reporting data disaggregated by institute or LEA poses two major probi . ns. First, processing 
and editing disaggregated data are fonnidable tasks, requiring substantial resources for the 
federal a ^ency with the responsibility for these tasks. Second, at the secondary level, there has 
been stiff resistance by many states to routinely reporting large amounts of data below the state 
level. For these reasons, VEDS relied, without success, or the states to perform the necessary 
error checking and editing. Reduced data burden would probably reduce resistance to 
disaggregated reporting. Hut some states object simply on general principle, expiessmg the view 
that the federal government has neither the need for nor the right to local data. 

In our view, this census should be limited to collecting data on program completers. Using 
data from the longitudinal s' 'es, it is possible to develop algorithms for estimating vocational 
education enrollment. How v political pressure for directly collecting some enrollment data is 
likely to continue to be strong. If so, we recommend limiting the coliectir-^ cf enrollment data to 
four general categories of vocational education: 
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• Industrial Arts; 

• Consumer and Homcmaking; 

• Other Occupationally Specific Vocational Education Programs in Grades 1 1 and above; 

• Other Occupationally Oriented Vocational Education Programs; 

At the secondary level, it is unlikely that much duplicated enrollment would occur within any 
one of these four categories — i.e., it is unlikely that a student would be taking two industrial 
arts courses simultaneously However, there would probably be significant duplication among 
the Tour categories — i.e., it is possible that some students would be taking a course in consumer 
and homemaicing and a course in business simultaneously 

Duplication among these categories, however, docs not constitute a seriouc problem. Just 
as it is useful to know how many sf.-^ents are enrolled in math, science, English, and foreign 
language courses in a given year, it is useful to know how many students are enrolled in each of 
these foui types of vcx:ational offerings. Tnst as it would be inappropriate to add together the 
number of students enrolled in math, .ce, and English, it would be inappropriate to add 
together students enroUf^d in these four categories of vocational education. The longitudinal 
studies, not :he universal census, will be the appropriate source of information on tht 
unduplicated count of students served by vocational education. 

At the postsecondary level, the potential for duplication within categories 3 and 4 is much 
greater, ay long as there are a substantial number of institutions without automated student record 
systems capable of routi ly unduplicating enrollment,^ Consequently, at the postsecondary 
level, it may be advisable tc seek data on contact hours, which is a superior measure of progra^n 
activity and a measure with which postsecondary institutions are more familiar than secondary 
institutions.*^ 

Whatever the merits of collecting this kind of general enrollment data in addition to 
completers, collecting detailed enrollment data is clearly unnecessary. The completion data 
collected from the universal census, in combination with completion and enrollment (kta from the 
longitudinal studies, can be used to develop algorithms that can be used to generate acceptable 
estimates of program enrollment by race, sex, and handicapping condition. The assignment of a 
program completer to a particular vocational program is unambiguous, while the assignment of 
student enrolled in a particular course to a specific program can be highly arbitrary and subject to 
considerable inconsistency. Hence, the error resulting from using completion-based algorithms 
to estimate program enrollment characteristics will be substantially less than the error that will 
result from attempting to develop program characteristics directly from course enrollment. On 
this point, the evidence from past experience with VEDS data is overwhelming. 

This emphasis on program completion is not merely a matter of methodological 
convenience. Concentrating on obtaining detail on completers is conceptually superior to an 



^Most postsecondai^ .. ititutions appear to have the capability to unduplicate counts of students within program 
categories; however, whether the number lacking such capability is sufficiently large to distort nation: . reporting 
is unknown and needs funhe< investigation. 

^Moreover, fhc secondary level there is a much closer ccMrcspondence between enrollment and contact hours in 
any one ct rhr four catCe;ories, because students are unlikely to be taking more than o- course in .^ny of the 
categories ?: any one time. 
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approach that seeks to develop detail on enrollment. The critical fact for policy making — 
whether it is concerned with labor market planning or with the distribution of marketable skills 
by race, sex, and special need — is the number of students prepared to obtain and hold entry 
level positions in jobs related to their training, which is, by our definition, the number of 
students completing a particular program. Mere enrollment in a program tells policy makers very 
httle about the likelihood of employment or the effectiveness of efforts to ensure that the 
acquisition of marketable skills is free of bias by race sex, and special need. In short, 
completion provides a critical measure of program performar^cc and is the indicator that needs to 
be reported as accurately and consistently as possible. 



DATA ON HANDICAPPED STUDENTS 

Section 423 of the Carl Perkins Vocational Education Act requires collecting data on 
secondary handicapped students enrolled in vocational education. Specipcally, Section 423 
states: 

The Secretary shall assure th .t adequate information on the access to vocational 
education programs by handicapped secondary school students be included in the 
national vocational education data system, required by section 161 of the Vocational 
Education Act of 1963 and by thi: part, for the biennial survey. The information base 
for the biennial survey for the handicapped shall be in 4-digit detail as defmed in A 
Classification of Instructional Programs published by the National Center for 
Education Statistics. The survey shall include information with respect to total 
handicapped enrollment by program, by type of instructional setting, and by type of 
handicapping condition. 

This section has two primary objectives: ^) obtaining reasonably accurate information on the 
status of handicapped students in vocational education ar J 2) improving communication between 
special educators and vocational educators to ensure that vocational education serves handicapped 
students effectively. 

The approach developed above to collect annual, universal data on program completers 
and, if necessary, enrollment meets both of these aims. Completer data would be reported by 
handicapping condition, and if enrollment datas is collected, providers could be asked to report 
the number of handicapped students by instructional setting.* Algorithms could be used to 
estimate handicapped and non*handicapped enrolhnent in the forty to sixty program areas. 

Such an approach should produce more accurate estimates of enrollment by program than 
would be achieved by "Attempting to collect enrollment data directly. Accurately assigning 
handicapped students to speciHc programs is as difficult as assigning non-handicapped students, 
perhaps more so because handicapped students are often more likely to be enrolled in less 
advanced, introductory courses that are not uniquely identified with a four-digit CEP code. In 
addition to greater accuracy In counts of handicapped students enrolled in vocational education, 
this approach has the advantage of generating valuable information on completers by 
handicapping condition, a data request that woula be difficult to justify if the information on 
handicapping condition were quested for enrollment. 



^Because a student compleung a program may have taken courses in a variety of instructional settings over ihe 
sequence of courses required for compleuon, it is not feasible to attempt to collect information on instructional 
setting for completers. 
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SPECIAT. STUDIES 



With longitudinal sample survevs collecting the detailed information on program 
enrollments and program outcomes and with a universal census collecting data on program 
completers, the major data requirements for policy puiposcs can be satsifed, ,;ith one important 
exception — finance. In our view, a thorough review of vocational education finances needs to 
be done only once in each reauthorization cycle, and we recommend that this become a special 
study. The Office of Vocational and Adult Education undoubtedly will need some annual 
accounting of federal funds, mcluding allocations by states to eligible recipients. While such 
information is necessary for accountability, it has little usetulness for policy. It cannot produce a 
complete picture of vocational education spending because the reporting requirements will apply 
only to recipients of federal funds. A special study v/ould address this larger policy concern. 

In addition to finance, there are other important topics that special studies could address. 
We need a better understandinjg of how states define and identify handicapped, disadvantaged, 
limited-English, and other special populations. Additionally, identifying and analyzing exemplary 
programs serving special populations would be useful. 

Another important concern is improving our knowledge of how vocational programs differ 
among different delivery systems — i.e., comprehensive high schools, vocational high schools, 
area vocational schools, community colleges, other postsccondary institutions, and proprietary 
schools. There is growing concern about the relative quality of vocational education programs in 
comprehensive high schools. While many, including ourselves, have expressed the view that 
programs are generally superior in vocational high schools and area vocational schools, these 
views are based mainly on personal observations rather than systematic study of the issue. 

Facilities are another topic suitable for special study. Many st^'-'; have used VEA funds for 
construction of new facilities, especially area vocational schools and vocational-technical 
institutes, and Congress continues to be concerned about the quality of buildings and equipment 
and about the geographic location and accessibility of facilites. As demonstrated by a survey 
conducted for tlie Department of Education by Westat in 1978, determining the location of 
vocational education facilities by type (comprehensive high school, vocational high school, area 
vocational school, community college, and so on) is straightforward and can yield some useful 
general information about accessibility. Collecting good data on the qvlity of equipment and 
facilities, however, remains highly problematic. 

In conclusion, we believe the reconmiendations proposed here would greatiy improve the 
quality of vocational education data at the federal level. Moreover, they are consistent with our 
pennissive planning model that focuses policy planning and evaluation on measures of program 
peformance. However, both permissive planning and the procedures recommend^.d for 
improving planning information represent a major departure from past practices. What are some 
of the problems that can be expected if they were implemented? To find out, we conducted 
several she visits to local secondary and postsecondary districts with vocational education 
offerings. Their reactions an* the subject of the next chapter. 
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CHAPTER SIX 
THE LOCAL PERSPECTIVE 



Local cooperation is critical to the success of pei.^'ssive planning. Unless local vocational 
educators understand and accept the methodology, cnteria, and measures used to evaluate their 
programs, the information they provide to the state and federal governments is likely to be of 
such low quality that it will he impossible to ev^uatc the programs meaningfully. To he)p avoid 
this problem, we made a significant effort to solicit the local perspective during the early stages 
of our efforts to design a sensible approach to performance -based evaluation. 

To do this, we made one-day site visits to five secondary and six post^econdary 
(community college) districts in northern California to discuss evaluation in general and our 
proposed approach in particular These districts were chosen randonily from a sample stratifed 
by geographic location, subject to tiie constraint that a selected district be no more than 150 miles 
from the San Franciso Bay Area in order to nJnimzc travel costs. W'^ had two main objectives: 
first, to determine what kinds of evaluation systems, criteria, and specific measures are 
acceptable to local administrators and teachers; and second, tc develop some preliminary 
conclusions about how implementation of the , pe of evaluation system we are pioposing should 
proceed. 

At the el^^ven districts visited we interviewed a toial of 16 directors or deans of vocational 
education and 18 teachers. To obtain as broad a range of perspectives as possible, the sample 
included both large and small urban, suburban, and rund districts. 

It should be stressed that we visited these districts early on in the project and that we went 
to them with a full-blown model of performance-based evaluation, similar to that described in the 
Appendix to Chapter Four. We had not yet settled on a completion-based system for annual 
evaluation. Indeed, our decision to opt for a more simple approach reflects much of what v,e 
learned through the site visits. Hence, the comments described here reflect respondents reactions 
to a more complicated system of evaluation than that recommended in Chapter Four. 
Nevertheless, the responses are quite useful in understanding the limits on state and federal 
evaluation if local coopera:ion is to be secured 

At the start of each intervi'jw we explained that we were trying to develop a new approach 
to state and federal evaluation that would hold local administrators accountable for the outcories 
of vocational education, and presented this as an alternative to the current practice of focussing 
on inputs or processes. We emphasized that the overall aim of this approach was tc give local 
administrators more flexibility in designing and operating vocationa? education programs so that 
they could rerpond to the needs of their communities. We then described the evzduation model in 
terms that were as non-technical as possible. The remainder of the interview was devoted to 
discussing the model and eliciting the local perspective on botn what v/e are proposing as an 
overall approach, and also on specific cnteria and measures for evaluatit g vocational education 
programs. 

The interviews revealed clearly that while there is genera^ agreement on the overall criteria 
upon which vocational education programs should be evaluated, there is no consensus on how 
evaluation should be conducted or on what specific measures should hf used. e discovered a 
wide range of opinions not only among but also within institutions. 'W^at follows is a suinmary 
of the various perspectives presented to us by local administralors and teav'^hers. Where 
appropriate, we have pointed out the implications of these views ftr implementation of a state 
and fedei^ evaluation system. 
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REACTION TO THE MODEL 



The reactions to the model variei /reatly, ranging from stong support to outright rejection 
of both the model and the very notion of state and federal evaluation of locally administered 
programs. In this section we summarize the various viewpoints, who held them, and why. 

Support for the model came from vocational educators who shared the belief that there 
exists " definite need for a simple, effective tool for assessing program outcomes. They took the 
position that vocational education programs should impart specific skills to students to prepare 
them for entry-level positions, believed that it was legitimate to develop measures to hold 
vocational educators accountable for doing so, and acknowledged that it is not now being done 
adequately. 

The supporters of the model were most disposed to favor it for programs governed by 
industry standards or preparing students to obtain statewide licenses or certificates, such as the 
health-related programs. In these types of programs, the curricula are necessarily similar from 
one institution to another, and there is general agreement on what needs to be taught As a result 
it was not hard for many to accept the idea of comparing the relative effectiveness of these 
programs. On the other hand, even the most enthusiastic supporters of the model were 
considerably less sanguine about its possibilities for evaluating programs such as merchandising 
or food management where the pro-am content varies greatly, and where there is disagreement 
among those in the field over what students shouM be taught This strongly suggests that any 
attempt to implement a full blown model of evaluation, especially on^ concentrating on 
employmwnt outcomes., should start with programs with industry standards, and only when 
acceptance of the mod^i for these programs ^as been achieved should the more diverse programs 
be tackled 

While some of those interviewed had philosophical reasons for suppor * % the model, 
others had more pragmatic ones. Vocational education, particularly in the com], /'ensive high 
school, is often seen as the underdog compared to the academic program, and some vocational 
educators saw the model as a potential tool to help them at the bargaining table at resource 
allocation time. A number of teachers and administrators complained that it was impossible to lun 
a first-rate program without up-to-date equipment, but that they faced constant uphill battles to 
get what they needed. Some felt that if there were specific standards that programs had to meet it 
would be easier to obtain adequate funds. 

Getting sufficient resources to do a good job has been a special problem for progiams in 
"eas with rapidly changing technologies. For example, one auto teacher pointed out that many of 
the diagnostic tools used in auto shops now are computerized. They are verj- expensive, but 
unless programs have them, they cannot produce job-ready students. If there were statewi-ie 
standards requiring that auto programs have certain equipment, the institutions 'vould be more 
easily persuaded to allocate funds for it (This could backfire on the teacher, however. The 
institution might decide the program was not worth it and close it down entiiely. Evjn ibis 
were a wise decision, it certainly would not please the auto teacher, who would probably 
immediately become an ardent foe of state standards.) 

The view that standards woulc' help raise the prestige of vocational education aid atteript to 
obtain more resources was most commonly found among high school administrators, but there 
were also some conmiunity college administrators who felt that vocational education v/^s plvi^n a 
lower priority in ^heir mstitutions than academic programs. One college administratoi complamed 
that deans in academic areas were always trying to find ways to get vocati^-^aal education dollars 
redirected He was therefore grateful for federal requirements that Hmit?d their ability to succeed, 
and would support standards that helped accompUsh tne same end. O'iier community college 
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administrators, however, felt no such threat. It appears, therefore, that the strongest support for 
program standards and evaluation may come from institutions in which vocational alucation 
programs feel the most vulnerable to attack or to loss of resources. 

Objections to the model, as in the case of support of the model, were sometimes 
philosopnical and sometimes pragmatic. A number of administrators, particularly at the 
community colleges, were hostile to the very notion of state and federal evaluation on the 
grounds that it violates the principle of local autonomy. They did net believe outside evaluation 
of program effectiveness to be legitimate or necessary, insisting instead that evaluation should be 
left to teachers, advisory committees, or local administrators. Many times we were told that it is 
the responsiblity of the local institution, not the state or the federal government, to decide 
whether or not a program is doing well. To those holding this view, a local institution needs to 
be accountable only to its community, not to any higher level of government, for the qualit>' of its 
programs. 

In part, many of those objecting to the model did so on the fear (justified) that it might lead 
to the cutting back, or even elimination of somf programs — that is, that programs that scored 
poorly would be pressured or forced by the state to contract or close down. This was seen to be 
in direct conflict noi only with the right of local institutions to decide what to offer, but with the 
smdents* rights to study in their own communities. 

This refusal on the part of some lo accept the legitimacy of state anc federal evaluation of 
local programs would no doubt cause serious problems if a complicated model of program 
evaluation ;/^re instituted on a universal annual basis Local administrators could easily sabotage 
the model by providing inaccurate data (as one of the administrators interviewed freely admitted 
to doing right now in meeting federal reporting requirements). Although it is difficult io know 
how widespread th^ opposition to state and federal evaluation is, we can be sure that there will be 
opponents to any evaluation system proposed. In implementing any evaluation system, then, 
attention must be paid to how reluctant participants ran be persuaded to provide timely, accu'-ate 
data despite their opposition. A variety of strategies arc available to do this, including involving 
local personnel at the planning stages, allowing local options (such as the fomut for transmitting 
data) wherever feasible, providing generous technical assistance, and devoting serious attention 
to dcvelopLig as many "side benefits" as possible so that the information generated for ti.^ state is 
a^so useful at the local level Above all, the reporting burden must be minimized. 

In addition to those who enthusiastically supported Ihe model and those who were 
unalterably opposed to it were a number of administratois an J teachers (probably the majority) 
who accepted the legitimacy of state and federal evaluation, but had various practical concerns 
about the model we were proposing. One major concern was the amount of work they expected it 
to involve on their parts. Vocational education administrators tended to feel already burdened 
with evaluation requirements and were reluctant to appear supportive of anything that would add 
to their workload. Community college administrators pointed out that in addition to collecting 
data for VEDS (a requirement for secondary districts as well but temporarily suspended), they 
must also meet the requirements of COPES (a time-consuming evaluation of one fourth of all 
vocational education programs each year) and SAM (a student accounting system for identifying 
vocadonr'^ education students). The follow-up of completers and leavers required by VEDS for 
both secondary and postsecondary institutions is seen as still another a considerable burden. 

Those opposing additional work did not necessarily rule out the model altogether. One 
administrator interviewed said he would be v. illing to adopt our approach if all the other 
requirements were dropped. Another indicated a willingness to accept program evaluation if it 
were part of the accreditation process. Another, resenting the emphasis on evaluation in 
vocati jnal education compared to other programs, suggested that he might be more amenable to 
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the type c/ evaluation v/e were proposing if a emic programs were subjected to the same 
scrutiny. 

Another major practical ^oncem raised in the interviews was the feasibility of making fair 
comparisons among programs with the same titles but with very different curricula. In practice, 
this problem would be much more serious in some areas than others. For example, in programs 
where there is state licensing, Cf>mparisons would be relatively easily to make because the 
programs art designed to ensure diat their students pass these examinations and therefore have 
similar curricula. In otlier programs, such as agriculture, merchandising, or office work, what is 
taught is more dependent on the needs of the local economy than on a state standard and this 
makes comparisons more difficult. This is a valid concem, and a difficult problem to solve. It is 
more easily discussed, however, in tlie context of specific outcome measures and therefore will 
be returned to below. 

Another aspect of the concem about the feasibility of making comparisons concerned taking 
into accou the fact that some programs have more resources, better equipment, and students 
who are easier to teach. While our model does try to take into account at least some of these 
contextual factors, some of those interviewed remained skeptical that our efforts could capture all 
of the important ones in a quantitative maimer. 

Finally, some administrators were concerned about possible misinterpretation of results. 
This is not an unreasonable concem. All too often the qualifications are not carefully presented or 
read when quantitative results are reported. This is a matter that has to be given serious attention 
by those responsible for administering the evaluation system, but in and of itself is no reason to 
abandon ev^duation. 

While we have to be very careful about making generalizations because of the relatively 
small number interviewed, the secondary district vocational education adminidtrators were, on 
the whole, the most positive about the potential of the model, the community college 
administrators the most negative, and the teachers relatively indifferent This is not particularly 
surprising. With as many as three quarters of the students enrolled in at least one vocational 
education course, vocational education is in a relatively strong position in the community 
colleges. As a result, communit>' college administrators were much less likely than high school 
administrators to need the model as a way of helping them out of an "underdog'" position. In 
addition, local autonomy is a particularly sensitive issue in California's community colleges. The 
increasing centralization of du:ision-making at the state level in recent years has made many 
community college administrators extremely wary of new proposals likely to lead to further state 
involvement Local autonomy is highly valued and guarded wherever possible. In the secondary 
districts, these feelings do not appear to be as strong, possibly because there is a longer tradition 
of state involvement in school management issues. 



Reaction to Specific Outcome Measures 

Despite the variety of opinions on the model and on state and federal evaluation in general, 
rioae of those interviewed disagreed with the criteria we i iggcstcd for evaluating vocational 
education pograms. While they might differ among themse^^'cs on how they would prioritize 
them, all sec .icd to accept employment, employability, edu . ' n, and access as legitimate and 
important outcomes of vocational education programs. What loilows is a summary of their views 
on the appropriateness and feasibility of various outcome measures. 
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1. Employment Outcomes 

There was general agreement on what constituted an "employment" outcome: placement in 
a job using the vocational training the student received Some teachers and administrators felt that 
any employment would meet this criterion, but some opposed counting unskilled jobs such as 
working in fast food establishments. Still others expressed the opinion that simply being hired 
for a job wa^* not enough, that demonstration of ability to hold a job was more indicative of a 
successful jutcome. Those holding this position, however, were quick to poinl out that the 
reason for leaving a job was important. While they were willing to be held accountable for 
students being able to keep jobs, they wanted to make sure that diey were not held accountable 
when a student left for personal reasons. 

Despite these qualifications, no one disputed the importance of placement Eacement seems 
generally accepted as the "bottom line" in vocational education. In other words, for a program to 
be successful, students 1 to get jobs. The community college administrators seemed to be 
relatively satisfied with using placement as the major criterion for evaluating vocational education 
programs. Most seemed to believe that if economic conditions were favorable and they could not 
place their completers, something was seriously wrong with the program. 

The high school administrators, while acknowledging the importance of placement, were 
somewhat less willing to accept it as the primary indicator of success. We were often ren'inded 
that vocational education at the high school level has important goals other than job placement, 
and that often students try out occupations and then decide they zxt not suited for them. 
Meanwhile, they have acquired basic educational and general work skills that will serve them 
well no matter what they decide to do. Vocational educators consider this a successful outcome 
even though there was no placement in a field related to the student's specific training. They did 
agree, however, that if there are not jobs in an area they should not be offering training for that 
occupation. 

The major difficulties respondents had with using placement to evaluate vocational 
education programs were not philosophical, but practical. Some maintained that placement alone 
does not ^ /e enough information to evaluate the intrinsic merits of a program or to compare it to 
others. As many of those interviewed for this study reminded us, placement depends on factors 
other than the quality of the program, such as the characteristics of the student and the prevailing 
economic conditions. Unless these are taken into account, comparisons are unfair and can lead to 
undesirable practices such as "skimming.'' One administrator ad^nitted that the demand that a 
particular JITA program produce an 80 percent placement rate had caused them to raise entry 
requirements to make sure that they admitted students likely *o succeed. This was achieved at the 
expen:;e of access of students with special needs to the program. 

Additionally, many of those interviewed stated repeatedly that collecting good follow-up 
data is expensive. Ideally, follow-up data should yield information not only on immediate 
placement in a job, but also on wages, advancement, stability of employment, and employer 
assessment of the employee's skills. This requires long term follow-up, important, but much too 
expensive to do on a regular basis at the local level. 

Even the more modest goal of contacting students once after they have had time to get jobs 
is expensive and not very cost effective. One administrator reported spending $2,300 for 
printing, mailing, and 1 oulating the returns from 3,000 forms. In spite of this investment, a 
response rate of only 20 percent was obtained for the first mailing. A second mailing brought it 
up to only 33 percent, far below the level considered necessary by survey researchers to draw 
valid conclusions. In other words, the $2,300 was simply wasted. 




This experience appears to be common. The response rate statewide to the follow-up 
surveys sent out to meet VEDS requirements in i981-82 was only 27 percent in the community 
colleges. While much higher izi the secondary districts, the secondary response rate of 69 percent 
was just barely adequate for making statistical estimates with any confidence. Employer surveys 
to evaluate the quality of the training of their employees have been even less successful. All those 
inteviewed said that is has been hard to get former students to identify their employers, to get 
employers to respond, and to compare the data received from different employers. 

In sum, while the administrators and teachers interviewed regarded employment outcomes 
as important measures of program sucess, not one of the institutions visited actually had follow- 
up data that were of sufficientiy high quality tc use to evaluate programs in a systematic way, nor 
did they have plans to try to obtain such data. Instead, they expected to continue to rely on 
informal information from students and local employers. While they did not suggest that this was 
equivalent tc systematic sxirveys, many felt that it did enable them to get a good idea of how & tr 
former students were faring, and thus, indirectly, the quality of their programs. 



2. Educational Outcomes 

There was a complete consensus among those interviewed that vocational education 
programs should include instruction in basic academic subjects, even at the remedial level if 
necessary. One administrator expressed the opinion that educational outcomes were as important 
as job-specific skills. She pointed out that reading and math skills are very important for 
employment, and that employers often requested ""students with basic skills'' rather than students 
wiiii specific job skills. She was also of the somewhat pessimistic view that vocational education 
programs were never going to have ""state of the art'' equipment in high tech occupational 
programs, which meant that employers were always going to have to train students. The 
implication of this, she felt, was that basic educational skills would be what the students had to 
offer. 

Despite the acceptance of the importance of basic academic skills, there was no agreement 
on how they should be taught. Some thought they should be integrated into occupational 
training, while others felt that they should be taught in separate departments, especially in the 
case of remedial progrants. It was argued that it takes special skills and training to teach remedial 
programs and that many vocational education teachers felt inadequate to the task. Others felt that 
stuiknts learned best if academic subjects were part of their vocational courses. 

One concern raised in the interviews was that students start at very different levels of 
academic competency when entering a program, making it difficult to compare programs in terms 
of either absolute Lkill levels or change in skill levels. A student entering a vocational program at 
a community college with a B.A., for example, would probably not show any significant 
improvement in reading level during the time spent in the program. Such a student would, 
however, be able to satisfy reading competencies required for completing the program, although 
it would not be the program perse that was responsible for achieving this accomplishment. 

Whe ^htd for opinions on how educational achievement should be measured, some of 
those interviv ved were amenable to the use of standardized tests administered pre and post 
program, while others felt that a final exam or program-related test would be fine. A few i.iought 
it unnecessary to test a'^hievement separately in academic areas to evaluate vocational education 
programs. 




3. Employability Outcomes 



Employability was a major focus of discussion in the interviews, and there was more 
controversy over this aspect of vocational education outcomes than any of the others. Everyone 
thought that employability was important, but not everyone agreed that it needed to be considered 
separately from employment. Some argued that if a person is employable, he or she should be 
able to find a job, and if a person finds a job, he or she obviously employable. A study of 
placement rates should, then, capture the effectiveness of programs in improving students' 
employability. 

The community college educators were less inclined to believe that it is necessary to 
measure employability separately. This was not surprising given their general satisfaction v ith 
placement as a criterion for evaluating programs. The administrators and teachers in programs 
with very hign placement rates were also less likely to be concerned about treating employability 
separately. Obviously, if they place all of their graduates, as some do, then the need to prove 
employability is not critical. 

High school administrators tended to be more enthusiastic about making an effort to 
Sep? :e employability from employment. A number cf thim pointed out that students often 
choose careers other than the one trained for, but that they still Ivam something valuable while in 
the program, even if they eventually drop out or switch to something else before completion. The 
administrators would lil^ to see this counted as a successful outcome. 

Among those who favored a separate treatment for employability, there was still 
disagreement on how to define and measure it With resi>cct defining employability, what was 
issue was whether or not it is possible to describe what makes a person employable. The general 
opinion seemed to be yes, where there are industry standards, but no for other areas. It was 
commonly believed that for Tcidxiy occupations a group of professionals would disagree as co 
what the standard practices and skills needed for a particular industry are, especially for 
occupations where there is a great deal of variety in equipment (word -^roccssing, for example) or 
organizational processes (merchandising, for example). A number those interviewed stated, 
quite simply, that "it just can't be done.** 

It was interesting, however, that there was little objection to the use of standards in fields 
already regulated by the state. The concern centered on the possibility of being forced to accept 
statewide standards that would not reflect what was currently being taught It was feared that this 
would lead to a narrov/ing of the curriculums and too much "teaching to the teat.** The fact that 
no concern was expressed about cither narrow curricula or teaching to the test in programs that 
have state standards now suggests that the real concern is not standards per sc, but the threat to 
•ocal autonomy implied by statewide standards for all occupations. If standards could be set 
locally, vocational oiucators might be much more willing to consider their use. 

Several teachers did, in fact« express a willingness to be held accountable for standards set 
locally. They felt that setting standards h d exp^tations for each program was an integral part of 
good teaching and that it was reasonable and fair to hold teachers accountable for meeting them 
(assuming the types of students enrolled and the resources available are taken into account) 

respect to measuring employability, many of those interviewed were at a loss to 
suggest ^ methodology. Of those who did have ideas, some favored informal methods, such as 
talking to teachers or advisory committees that included local employers. A number of 
administrators, in rural areas particularly, tavorcd the latter approach. They felt that they had 
good contacts with local employers, and were confident thai if their programs were not 
producing employable students they would hear about it Others relied on indirect indicators such 
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as student demand or the number of employers coming to campus to interview prospective 
employees. 

In short, administrators felt that they were on top of the situation at their mstitutions and 
knew what the good programs were through informal evaluation methods. This may be true, but 
this attitude points out the difference between the perspectives adopted by state and local 
administrators. These informal methods do not allow systematic comparison of programs among 
institutions. While they do provide valuable information that should be integrated into the overall 
evaluation system, alone they arc not sufficient 

A number of community college administrators mentioned accreditation processes as a way 
of gaining information on the quality of programs and thus the employability of their students. 
All colleges arc accredited every five years, and some programs arc accredited individually. Most 
nursing and other health science programs arc accredited, for example. This accreditation is 
voluntary, and is carried out by state officials or professional associations, usually in fields 
where there is a state licensing test The acotditation of a program includes a thorough self-study 
and a visit from a team of outside professionals. The review lasts about three days. To become 
accredited, schools must fulfill certain requirements regarding equipment, teacher-student ratio, 
curriculum offerings, and so on. Some administrators felt that this thorough, unbiased review by 
outside professionals was a valuable way to evaluate programs and thus the employability of 
their students. One administrator commented that she preferred this type of review to advisory 
committee reviews because members of the advisory committee are often friends of the 
instructors and because advisory committees rarely spend much time actually observing the 
programs. 

We suggested, as an alternative to these methods of assessing employability, the 
administration of competency tests to determine whether or not students have mastered specific 
skills necessary to obtain jobs related to their training. As might be expected, we found both 
proponents and opponents of such a system, and many of the proponents qualified their support. 

Those who accepted the idea of standards for vocati il education programs liked the idea 
of competency tests. They did, however, have different ideas about how Uiey should be usai and 
what they should cover. Some thought that the competencies should be simple and relatively easy 
to achieve, while others preferred that they be more detailed and difficult Some instructors felt 
that the state licenses in brakes and headlights and in emission control were valuable tools for 
assessing students' skill levels. Other instructors, however, saw these tests as a bare minimum 
that did not reflect the skill levels they wanted their students to obtain. One auto mechanics 
teacher said that 85 percent of his students could pass the state license test after only a short time 
in his program, but that they would not be ready to work in a shop that soon. 

A similar reaction was found in the health field, where most students pass the state 
licensing tests on the first attempt Some perceived the tests as valuable tools for assessing 
perfomance, but a number of others said the skill levels were too low to be meaningful. They 
pointed out that, with over ninety percent passing the test, the results would be of limiteu utility 
for making comparisons among programs in different institutions. 

The question of whether to adopt minimum competencies or to design ? multi-level system 
of competencies is an important one. Some of those interviewed felt that in industries where there 
are not uniform standards the only viable standard was a minimum one. Instructors in some 
institutions, however, were bitterly opp^^^d to minimum competencies. They claimed, on the 
basis of past experience, that they have u ^ often seen the minimum become the standard or the 
average. They maintained that students are very often unwilling to go beyond the minimum level 
of certification. They therefore strongly supported a ladder of competencies, where students 
could move from one level to another as their skills incrcased. 
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Some of those favoring competency testing in principle were skeptical that appropriate tests 
could be devised. There was general agreement that "paper and pencil" tests would not suffice in 
most cases, and that various types of "hands on" tests would have to be developed in place of or 
to supplement them. One high school administrator suggested a two-tiered test. The first test 
would be a written test to determine whether or not the student had mastered basic concepts in the 
field and had acquired basic knowledge about equipment and occupational safety. A teacher in an 
automotive program suggested that the first test might examine the student's understanding of 
basic shop safety, major types of engines, how the various systems of the automobile work, and 
how they work together. The second would be a practici one where the student would be 
presented with specific tasks to complete, such as diagnosing an automotive problem or 
operating computer equipment 

Interestingly, some institutions have been experimenting with proficiency testing. Two of 
the districts visited award certificates of proficiency and a third was considering them. In one 
place, certificates of completion are awarded to students passing the required courses and a 
certificate of proficiency to those sludrnts who demonstrate entry-level skills in the occupation. 
All of the persons using or considering using certificates were quite positive about them. They 
felt that they gave students someining to work for and a feeling of accomplishment when 
achieved They also felt they are useful for employers because they provide concrete information 
on the training the students were exposed to, Finsdly, they felt that the certificates make teachers 
more accountable for their performances. When a teacher signs a certificate, he or she is 
certifying that the student actually has certain skills. Perhaps, then, locally-developed standards 
might be acceptable for at least some areas where statewide standards are rejected. 

In sum, vocational educators and administrators at the local level believed in th : idea of 
employability as an important outcome. Community college people, more satisfied with 
placement as a criterion for evaluating programs than secondary district people, were less likely 
to consider it necessary to separate cmployability from employment outcomes. Of those who 
would like to measure ernployability separately, most were only willing to accept competency 
tests in fields for which industry standards exist. The idea of statewide standards in other areas 
was greeted with apprehension. The enthusiasm with local proliciency certificates where used, 
on the other hand, suggests that locaUy-developed standards might be more readily accepted. 



4. Access Outcomes 

The idea that vocational educators have an obligation to try to increase the access of 
disadvantaged, handicapped, and limited English proficiency students to vocational education 
programs and of both men and women to ^.rograms nontraditional for their sexes was universally 
accepted and generated little discussion or controversy. All agreed that access should be 
monitored. Unfortunately, at the time of the site visits, we had not vet settled upon the "access 
ratio" as the measurr of access, nor had we developed its use in the allocation of funds. 
Consequently, we could not discuss these aspects of the proposed evaluation/funding system 
during the site visits. 



CONCLUSIONS 

The site visits led us to a number of important conclusions that significantly affected the 
final design of our reconunended approach to performance-based evaluation. First, from the 
responses of many of those interivewed, it became clear that a complex model of program 
eviduation is not workable, as long as it must depend on local administrators to collect, maintain, 
and report the necessary data. Local administrators are already overburdened, and ir most cases 
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they have neither the financial resources nor the expertise to collect detailed data, especially 
follow-up data, required for a model systematically comparing the effectiveness of vocational 
education programs. When such data arc collected by an independent collector — such as a 
professional contractor charged with data collection for the longitudinal studies conducted by the 
U.S. Department of Education — they can be employed in a model of tne kind suggested in the 
Appendix to Chapter Four. It is important, therefore, that the design of longitudinal studies 
consider carefully data requirements for program evaluation. 

Second, local staff must be given considerable discretion in the development of the 
standards and competencies required to certify that a student completing a particular vocational 
education progi. m is sufficicnily prepared for entry level employment in a field related to 
training. This discretion is necessary not only because of the considerable variation in local 
economic conditions that exists among providers, but also because it i . the most likely to secure 
Ihe cooperation of local officials, many of whom, simply on the basis of strong philosophical 
beliefs in local autonomy, will resist any effort to impose standards "from the top down." 
Involvement by business, labor, and other interested parties in the defintion of stnadards and 
competencies will help to ensure that they are appropriate to local economic conditions. 

Third, the hallmark of any ongoing evaluation and reporting process required of all 
providers of vocational education must be simplicity. The enormous detail required by VEDS for 
data collection and reporting, as well as the ambiguity and confusion that surrounded the 
requirements for allocating and spending funds under the 1976 Amercements, have generated 
much needless work for local administrators whose time would be far better spent on oUier tasks. 
The mindlessness of some of these requirements have also produced among many teachers and 
administrators a general disregard for the wisdom of federal and state policy in vocational 
education. 

The system for performance-based evaluation proposed in Chapter Five and the changes 
recommended in Chapter Five for federal data collection have tried to incorporate the advice 
offered during the site visits. Our suggestions may not achieve the degree of simplicity and 
elegance that we would all like, but they do represent a significant improvement over past policy. 
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CHAPTER SEVEN 



FUTURE DIRECTIONS FOR FEDERAL POLICY: 
A CONCLUDING NOTE 



Aside from stimulating large program growth, federal vocational education policy has 
enjoyed few successes over the last 25 years. Throughout this period, vocational education has 
been under constant attack from one front or another for failing to deliver quality education and 
for failing to make it accessible to all. The latest salvo comes from the Committee for Economic 
Development, whose trustees represent 225 of the nation's leading corporations and institutions 
of higher learning: 

Many "vocational education" programs are almost worthless. They are a 
cruel hoax on young people looking to acquire marketable skills. So many 
different and, in many cases, unproductive programs in our public schools have 
been called "vocational education" that most existing programs need to be 
disbanded and reshaped. Vocational education should ensure that students are 
learning skills that relate to the real needs of the job maiket.^ 

Despite such criticisms, vocational education has remained remarkably popular with 
Congress. In the summer of 1985, amidst growing pressures from all sides to decrease the 
federal deficit. Congress voted an increase of $100 million in federal funds for vocational 
education, one of the few domestic programs to enjoy in^^reascd support ^ Why docs Congress 
continue to spend federal dollars for vocational education m spite of the overwhelming evidence 
of the past ineffectiveness of federal policies? 

One explanation, of course, is the power of the nation's education lobbies. The recent $100 
million increase for vocational education had the support not only of the American Vocational 
Association but also the National Education Association and Uie American Association of 
Community and Junior Colleges. But the strength of special interests is too pat an answer, one 
that not only overestimates the power of the lobbies but also fails to appreciate the complex 
appeal of vocational education. 

Congress continues to support vocational education, we suspect, because there is a strong 
consensus that the basic aim of vocational education — to prepare young people for the world of 
work — is essential to the well-being of the nation. Few would argue that this aim should be the 
only concern of schooling, but few would deny that it must be one of schools' major objectives 
if young people are to leiui satisfying, independent adult lives. To oppose vocational education, 
therefore, seems in a very primordial way to deny young people the means tc survive and 
prosper. 

To make vocational education work consistently, however, has proven to be exceedingly 
difficult No one will quarrel with the Committee for Economic Development's statement that 
"vocaiional education should ensure that students are learning skills that relate to the real needs of 
the job market" Does anyone seriously believe that most vocational educators have not been 
trying to do precisely that? The problem is that this is very hard to do well for all students, in all 
fields, at all times. What exactly arc the "real needs" of the job market? At what point in ti Tie? 



^Excerpts in Education Week. Vol. V, No. 2, September 11, 1985, p. 17. 
^Update. Vol 8, No. 1, August 1985, p. 1. 
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According to whom? What job market — local, regional, national, international? Precisely how 
should vocational education "ensure" that students arc learning appropriate skills? Until such 
questions can be answered more clearly, vocational education win continue to muddle along an 
inconsistent path of success and failure* 

In this monograph, we have tried to suggest scine ways that federal policy might be recast 
to help iO answer such questions. First and foi*nrst, we have argued that future policy needs to 
emphasize perfomumce over process. Prcscripi 'eral policy that is concerned mainly with 
how services are delivered is not only at odds wit* nation's decentralized approach to school 
governance but also tends to lose sight of what .^rvices should accomplish and why. Future 
federal policy should strive to be more pcrmisRive, encouraging states and local schools to define 
specific, measurable standards of performance expected from vocational education programs and 
rewarding them for producing students who meet these standards. These standards should be 
tailored to local circumstances and reflect balanced input from educatois, business, and labor. 

Second, we have also urged that the objectives, procedures, and scope of state and national 
data collection concentrate on issues of effectiveness rather than compliance. For policy to 
become performance-oriented, we need to know what woiics in vocational education, what does 
not work, and why. VEDS, which understandably was designed to evaluate compliance with the 
many prescriptive features of P.L. 94-482, cost millions of dollars and collected enormous 
amounts of data — none of which can tell us stny thing about what vocational education programs 
are effective, what the characteristics are of effective programs, what types of students have been 
most successful in vocational education, what types of delivery systems are effective for different 
kinds of students, and a host of other important questions on the performance of vocational 
education. In the absence of such information, it is virtually impossible to develop sound federal 
policy. 

Third, future policy needs to link concerns about access to performance. To date, federal 
policy has been satisfied with assurances that groups historically under represented in certain 
vocational education programs are now being adequately served, as evidenced by their 
participation in these programs. Participation, however, provides no guarantee that these students 
are in fact acquiring the skills they need to perform effectively in the labor maiket Only program 
completion, which certifies that students have acquired th«; basic and job-specific skills necessary 
to perform an entry level job in a field related to training, indicates that meaningful access has 
been achieved. 

Finally, future policy needs to tie pertbrmance and access directly to funding in a clear, 
simple fashion. From past experience with P.L. 94-482, there can be no doubt that complicated 
procedures for allocating funds are unworkable. The Carl Perkins Act eliminated the funds 
distribution requirements but unfortunately provided nothing in their place. Although there is as 
yet no information on what federal funds are buying under the Carl Perkins Act, it is likely that 
Congress will find once again that there is littie relationship between funding and the realization 
of f^eral policy objectives. 

Is it feasible to reshape federal vocational education policy to encourage the definition and 
adoption of performance standards, to collect data suitable fo; analyzing progr am effectiveness, 
to define access in terms of performance, and to base funding on program accomplishments? In 
some respects, the climate at the federal and state levels has never been better for pursuing ^uch 
changes. The renewed national concern about the quality of public education hab focused 
legislative attention on program standards and on securing other reforms in the daily operations 
of elementary, secondary, and postsecondary institutions. This renewed concern for improving 
curriculum and tlie quali^ of teaching provides vocational education with an opportunity but also 
a warning. 
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On the one hand, the new attention on curriculum and teaching provides an oppoitunity for 
a critical review of vocational education programs and their role in the larger secondary and 
posts econdary curriculum. It is an opportunity to state clearly what vocational education is 
seeking to accomplish and to develop fair, useful criteria for evaluating its effects. On the other 
hand, the latest pursuit of excellence in education is also a warning that unless vocational 
education can demonstrate both its ability to strengthen basic academics as well as its contribution 
to the acquisition of life-long skills that will outlast a.i entry level job, vocational education will 
find itself increasingly squeezed out of the mainstream secondary and postsecondary curriculum. 

Federal policy can help improve vocational education and ensure that it benefits from the 
nationwide push for educational reform, but to do so, it must abandon the prescriptive 
procedures of the past and focus on defining desirable program outcomes and rewarding 
programs that achieve the desired results. In this fashion, the twin federal aimo to improve 
programs and to improve access may finally be realized. 
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